Systems and Methods for Source Signal Separation

ABSTRACT

A method of processing a signal, including taking a signal formed from a plurality of source signal emitters and expressed in an original domain, decomposing the signal into a mathematical representation of a plurality of constituent elements in an alternate domain, analyzing the plurality of constituent elements to associate at least a subset of the constituent elements with at least one of the plurality of source signal emitters, separating at least a subset of the constituent elements based on the association and reconstituting at least a subset of constituent elements to produce an output signal in at least one of the original domain, the alternate domain and another domain.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. provisional patentapplication Ser. No. 61/749,606 filed Jan. 7, 2013, U.S. provisionalpatent application Ser. No. 61/785,029 filed Mar. 14, 2013, and U.S.provisional patent application Ser. No. 61/642,805 filed May 4, 2012 allof which are hereby incorporated by reference in their entirety.

BACKGROUND

1. Field of the Invention

The present invention relates to methods and systems for signalprocessing and, more specifically, to methods and systems for separatinga signal into different components.

2. Description of the Related Art

Signal separation (SS) is a separation of any digital signal originatingfrom a source into its individual constituent elements, such that thoseelements may be deconstructed, isolated, extracted, enhanced, orreconstituted in isolation, in part, or in whole. SS may be performed onany form of data including auditory data and/or visual data or images.SS may be performed using a plurality of source dependent methodologiesincluding principal components analysis, singular value decomposition,spatial pattern analysis, independent component analysis (ICA),computational auditory scene analysis (CASA) or any other suchtechnique.

Conventional SS techniques typically require prohibitive amounts ofprocessing to achieve real or near real time performance and are thusfar quite often incapable of effectively identifying and isolatingsignal sources within a given signal. There is therefore a need for asystem and algorithms for operating such a system that provides for realor near real time signal separation.

SUMMARY OF THE INVENTION

The methods and systems for SS in accordance with various embodimentsdisclosed herein are source-agnostic. The nature of the original signalis generally irrelevant with respect to generation methodology orapparatus. Signal sources to which SS systems and methods may be appliedinclude but are not limited to sound, audio, video, photographic,imaging (including medical), communications, optical/light, radio,RADAR, sonar, sensor and seismic sources. The methods and systemsdescribed herein may include a set of source agnostic systems andmethods for signal separation. These include methods of high-resolutionsignal processing to mathematically describe a signal's constituentparts, methods of tracking and partitioning to identify portions of asignal that are “coherent” i.e., emanating from the same source—andmethods to re-combine selected portions, optionally in the originalsignal format, and/or sending them directly to other applications, suchas a speech recognition system.

In accordance with an exemplary and non-limiting embodiment, a method ofprocessing a signal comprises taking a signal formed from a plurality ofsource signal emitters and expressed in an original domain, decomposingthe signal into a mathematical representation of a plurality ofconstituent elements in an alternate domain, analyzing the plurality ofconstituent elements to associate at least a subset of the constituentelements with at least one of the plurality of source signal emitters,separating at least a subset of the constituent elements based on theassociation and reconstituting at least a subset of constituent elementsto produce an output signal in at least one of the original domain, thealternate domain and another domain.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain signal comprises receiving an inputsignal comprising a time domain signal stream and creating a firstwindowed data set and a second windowed data set from the signal stream,wherein an initiation of the second windowed data set time lags aninitiation of the first windowed data set, converting the first windoweddata set and the second windowed data set to a frequency domain andstoring the resulting data as frequency domain data having a fundamentaltransform resolution, performing complex spectral phase evolution (CSPE)on the frequency domain data to estimate component frequencies of thefirst and second windowed data sets at a resolution greater than thefundamental transform resolution, using the component frequenciesestimated in the CSPE, sampling a set of frequency-domain highresolution windows to select a frequency-domain high resolution windowthat fits at least one of the amplitude, phase, amplitude modulation andfrequency modulation of a component of an underlying signal wherein thecomponent comprises at least one oscillator peak, using a trackingalgorithm to identify at least one tracklet comprised of one or moreoscillator peaks that emanate from a single oscillator source within theunderlying signal, grouping tracklets that emanate from single sourcesand providing an output signal.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain signal comprises receiving a timedomain signal stream and creating a first windowed data set and a secondwindowed data set from the signal stream, wherein an initiation of thesecond windowed data set time lags an initiation of the first windoweddata set, converting the first windowed data set and the second windoweddata set to a frequency domain and storing the resulting data asfrequency domain data having a fundamental transform resolution,performing complex spectral phase evolution (CSPE) on the frequencydomain data to estimate component frequencies of first and second windowat a resolution greater than the fundamental transform resolution, usingthe component frequencies estimated in the CSPE, sampling a set ofstored high resolution frequency-domain windows in a singlettransformation process to select a high resolution frequency-domainwindow that fits at least one of the amplitude, phase, amplitudemodulation and frequency modulation of the underlying signal oscillator,storing the parameters required for at least one of FM creation and AMcreation in the frequency domain, wherein the parameters for FM creationinclude amplitude, phase, reference frequency, and modulation rate andthe parameters for AM creation include amplitude, phase, frequency, andamplitude envelope information and recreating the frequency spectrum forat least one of an FM-modulating oscillator peak and an AM-modulatingoscillator peak, such frequency spectrum including any transient effectswhere the oscillator may turn on or off at some point within the datasample window.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a signal comprises receiving a plurality of signalstreams that may interfere with each other to some extent and creatingfirst and second sets of input sample windows wherein an initiation ofthe second set time lags an initiation of the first set, converting bothof the input sample windows from a time domain to a frequency domain,the resulting frequency domain data having a fundamental transformresolution, performing complex spectral phase evolution (CSPE) on thefrequency-domain data to estimate component frequencies of the first andsecond data sets at a resolution greater than the fundamental transformresolution, using the component frequencies estimated in the CSPE,sampling a set of high resolution windows to select a high resolutionwindow that, when properly multiplied by appropriate factors, fits atleast one of the amplitude, phase, amplitude modulation and frequencymodulation of an underlying signal component, using a tracking algorithmto identify at least one tracklet of oscillator peaks that emanate froma single oscillator source within the underlying signal, groupingtracklets that emanate from a single source, rejecting tracklets thatare likely to be associated with noise or interfering signals, selectingat least one grouping of tracklets, reconstructing a signal from theselected groupings of tracklets and providing an output in a desiredformat using the selected grouping.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a signal comprises taking an original signal formedfrom a plurality of source signal emitters, the original signalexpressed in an original domain, decomposing the signal into amathematical representation of a plurality of constituent elements in analternate domain, analyzing the plurality of constituent elements toassociate at least a subset of the constituent elements with at leastone of the plurality of source signal emitters, separating theconstituent elements based on the association and preserving theconstituent elements of the original input signal that correspond to atleast one desired signal source for use as an output signal in theformat of the original input signal.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a signal comprises taking a signal formed from aplurality of source signal emitters and expressed in an originaltransform domain, transforming the signal into a mathematicalrepresentation of a plurality of constituent elements in an alternatedomain, performing complex spectral phase evolution (CSPE) on thecombined alternate-domain data to estimate constituent elementcharacteristics at a resolution greater than the resolution of theoriginal transform domain, analyzing the plurality of constituentelements to associate at least a subset of the constituent elements withat least one of the plurality of source signal emitters, separating atleast a subset of the constituent elements based on the association andreconstituting at least a subset of constituent elements to produce anoutput signal including output from at least one desired source signalemitter in at least one of the original domain, the alternate domain andanother domain.

In accordance with another exemplary and non-limiting embodiment, amethod of separating components of an input signal comprises taking asignal formed from a plurality of source signal emitters that interferewith each other to some extent, the input signal expressed in anoriginal domain, decomposing the signal into a representation of aplurality of constituent elements in an alternate domain, representingthe decomposed signal in a unified domain data structure that allowsrepresentation of phase, frequency, amplitude, and directionalinformation, analyzing the plurality of constituent elements toassociate at least a subset of the constituent elements with at leastone of the plurality of source signal emitters, the analysis includinguse of a unified domain directional estimate to assist in theassociation, separating at least a subset of the constituent elementsbased on the association and reconstituting at least a subset ofconstituent elements to produce an output signal including output fromat least one desired source signal emitter in a desired format.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain signal comprises receiving a timedomain signal stream and creating a first window and a second windowfrom the signal stream, wherein an initiation of the second window timelags an initiation of the first window, converting at least one of thewindows to a frequency domain and storing the resulting data asfrequency domain data having a fundamental transform resolution,performing complex spectral phase evolution (CSPE) on the frequencydomain data to estimate component frequencies of first and second windowat a resolution greater than the fundamental transform resolution, usingthe component frequencies estimated in the CSPE, sampling a set ofstored high resolution frequency-domain windows in a singlettransformation process to select a high resolution frequency-domainwindow that fits at least one of the amplitude, phase, amplitudemodulation and frequency modulation of the underlying signal oscillator,storing the parameters required for at least one of FM creation and AMcreation in the frequency domain, wherein the parameters for FM creationinclude amplitude, phase, reference frequency, and modulation rate andthe parameters for AM creation include amplitude, phase, frequency, andamplitude envelope information and recreating the frequency spectrum forat least one of an FM-modulating oscillator peak and an AM-modulatingoscillator peak, such frequency spectrum including any transient effectswhere the oscillator may turn on or off at some point within the datasample window.

In accordance with another exemplary and non-limiting embodiment, amethod for detecting signal modulation comprises providing a set ofmodulating complex operators having real and imaginary parts that can beapplied to a stable oscillator to produce a modulated oscillator,providing a further set of operators known as pullback operators thatcan counteract the modulating effects on an oscillator, applying apull-back operator to a modulated signal so that when the signal istransformed from a time domain to a frequency domain at least a portionof a modulation effect is counteracted, applying a pull-back operator toa time-lagged sample window data set so that the modulation effect iscounteracted in the time-lagged sample window so as to revert it to astate that can be compared to the transform of the initial sample windowand comparing the transformed initial and pulled-back time-lagged samplewindow data to derive a super-resolution transform representation thatreveals the underlying reference frequency for the frequency modulationof the modulated signal.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a signal comprises receiving a plurality of signalstreams that interfere with each other to some extent and creating firstand second sets of input sample windows wherein an initiation of thesecond set time lags an initiation of the first set, converting thefirst and second sets of input sample windows from a time domain to afrequency domain, the resulting frequency domain data having afundamental transform resolution, performing complex spectral phaseevolution (CSPE) on the frequency-domain data to estimate componentfrequencies of the first and second data sets at a resolution greaterthan the fundamental transform resolution wherein the CSPE uses windowsizes of varying length, using the component frequencies estimated inthe CSPE, sampling a set of stored high resolution windows to select ahigh resolution window of a first window length that fits at least oneof the amplitude, phase, amplitude modulation and frequency modulationof an underlying signal component comprising a plurality of oscillatorpeaks, using a tracking algorithm to identify at least one tracklet ofoscillator peaks that emanate from a single oscillator source within theunderlying signal, grouping tracklets that emanate from a single source,rejecting tracklets that are likely to be associated with noise orinterfering signals, selecting at least one grouping of tracklets,reconstructing a signal from the selected groupings of tracklets whereinthe reconstruction uses windows of a desired length that is optionallydifferent from the first window length analyzed in the CSPE andproviding an output in a desired format using the selected grouping.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a signal comprises receiving a time domain signalstream and creating a first window and a second window from the signalstream, wherein an initiation of the second window time lags aninitiation of the first window, converting the first window and secondwindow to a frequency domain and storing the resulting frequency domaindata having a fundamental transform resolution, performing complexspectral phase evolution (CSPE) on the frequency domain data to estimatecomponent frequencies of the windowed data at a resolution greater thanthe fundamental transform resolution, using the component frequencies,sampling a set of stored frequency-domain high resolution windows toselect frequency-domain high resolution windows that may fit themodulation of a component of the underlying signal wherein the componentcomprises a plurality of oscillator peaks, using a tracking algorithm toidentify at least one tracklet of oscillator peaks that emanate from asingle oscillator source within the underlying signal and usinginformation from a behavior of the tracklet with which an oscillatorpeak is associated to assist in providing an estimate of the modulationof the oscillator.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a signal comprises analyzing the super-resolutionfrequency information in a sequence of windows of data, combining thesuper-resolution frequency information with the corresponding phaseinformation for the windows of data, modeling the evolution of thesignal over the windows of data to predict the frequency or phase of thesignal for windows that are advanced in time or backward in time, usingany combination of signal frequencies or phases to predict the expectedvalues for any set of frequencies and phases that are not included inthe model prediction.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a signal to produce a mathematical decomposition ofthe signal in such a way that the decomposed elements of the signal canbe recombined to produce a lossless representation of the originalsignal comprises creating a model of the signal in data windows using asum of oscillator peaks created using short-time stable oscillators,frequency modulating oscillators, and amplitude modulating oscillators,removing each modeled signal element from the original signal until adesired degree of accuracy is achieved and so that all that remains is asufficiently small residual signal, encoding the residual signal so thatit can be reproduced exactly and storing the parameters of theoscillator peaks used in the modeling of the data along with the encodedresidual signal so that they can be recombined into an exact losslessreconstruction of the original signal.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain signal comprises receiving a timedomain signal stream and creating a first window and a second windowfrom the signal stream, wherein an initiation of the second window timelags an initiation of the first window, converting the first window andthe second window to a frequency domain and storing the resulting dataas frequency domain data having a fundamental transform resolution,performing complex spectral phase evolution (CSPE) on the frequencydomain data to estimate component frequencies of the windowed data at aresolution greater than the fundamental transform resolution, using thecomponent frequencies, sampling a set of stored frequency-domain highresolution windows to select a frequency-domain high resolution windowthat fits at least one of the amplitude, phase, amplitude modulation andfrequency modulation of a component of an underlying signal wherein thecomponent comprises at least one oscillator peak, using a trackingalgorithm to identify at least one tracklet comprised of one or moreoscillator peaks that emanate from a single oscillator source within theunderlying signal grouping tracklets that emanate from a single sourceand providing an output signal.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain signal comprises receiving aplurality of signal streams and creating first and second sets of inputsample windows each corresponding to one of the plurality of signalstreams, wherein an initiation of the second set of input samples timelags an initiation of the first set of input samples, converting thefirst and second input sample windows to a frequency domain and storingthe resulting data as frequency domain data having a fundamentaltransform resolution, performing complex spectral phase evolution (CSPE)on the frequency domain data set to estimate component frequencies ofthe input sample windows at a resolution greater than the fundamentaltransform resolution, using the component frequencies estimated in theCSPE, sampling a set of stored high resolution windows to select a highresolution window that fits at least one of the amplitude, phase,amplitude modulation and frequency modulation of a component of anunderlying signal wherein the component comprises at least oneoscillator peak, using a tracking algorithm to identify at least onetracklet comprised of one or more oscillator peaks that emanate from asingle oscillator source within the underlying signal, groupingtracklets that emanate from a single source and providing an outputsignal.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain signal comprises receiving a timedomain signal stream and creating a first window and a second windowfrom the signal stream, wherein an initiation of the second window timelags an initiation of the first window, converting the first and secondwindows to a frequency domain and storing the resulting data asfrequency domain data having a fundamental transform resolution,performing complex spectral phase evolution (CSPE) on the frequencydomain data to estimate component frequencies of the first and secondwindows at a resolution greater than the fundamental transformresolution, using the component frequencies estimated in the CSPE,sampling a set of stored high resolution frequency-domain windows toselect a high resolution frequency-domain window that fits at least oneof the amplitude, phase, amplitude modulation and frequency modulationof an underlying signal component comprising at least one oscillatorpeak and removing the effects of the estimated component from at leastone of the stored windowed data sets, using a tracking algorithm toidentify at least one tracklet of oscillator peaks that emanate from asingle oscillator source within the underlying signal, groupingtracklets that emanate from a single source and providing an outputsignal.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain signal, comprises receiving aplurality of signal streams and creating first and second windows eachcomprising a set of input samples corresponding to one of the pluralityof signal streams, wherein an initiation of the second window lags aninitiation of the first window, converting the first and second windowsto a frequency domain and storing the resulting frequency domain datahaving a fundamental transform resolution, representing a plurality ofchannels each comprising a first set and a second set of frequencydomain data in a unified domain representation and performing complexspectral phase evolution (CSPE) on the frequency domain data to estimatecomponent frequencies of the frequency domain data at a resolutiongreater than the fundamental transform resolution of the frequencydomain data, including using the phase rotation measured between twotime-separated sample windows to detect an actual underlying frequencyat said greater resolution, using the component frequencies estimated inthe CSPE, sampling a set of stored high resolution windows to select ahigh resolution window that fits at least one of the amplitude, phase,amplitude modulation and frequency modulation of the underlying signalcomponent comprising at least one oscillator peak, using a trackingalgorithm to identify at least one tracklet of oscillator peaks thatemanate from a single oscillator source within the underlying signal,grouping tracklets that emanate from a single source and providing anoutput signal.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain signal comprises receiving a timedomain signal stream comprising an original signal and creating a firstwindowed data set and a second windowed data set comprising samples fromthe signal stream multiplied by an analysis window, wherein aninitiation of the second window lags an initiation of the first window,converting the first and second windowed data sets to a frequency domainand storing the resulting data as frequency domain data having afundamental transform resolution, performing complex spectral phaseevolution (CSPE) on the frequency domain data to estimate componentfrequencies of the first and second windowed data sets at a resolutiongreater than the fundamental transform resolution, using the componentfrequencies estimated in the CSPE, sampling a set of frequency domainhigh resolution windows to select a high resolution window that fits atleast one of the amplitude, phase, amplitude modulation and frequencymodulation of a component of an underlying signal wherein the componentcomprises at least one oscillator peak and reproducing a selectedportion of the original signal as an output signal.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain signal comprises receiving a timedomain signal stream and creating a first window comprising a first setof input samples (multiplied by an analysis window) in the time domainand a second window comprising a second set of input samples (multipliedby an analysis window) in the time domain from the signal stream,wherein an initiation of the second set of input samples time lags aninitiation of the first set of input samples, converting the first andsecond input sample windows to a frequency domain and storing theresulting data as frequency domain data having a fundamental transformresolution, performing complex spectral phase evolution (CSPE) on thefrequency domain data to estimate component frequencies of the first andsecond windowed data sets at a resolution greater than the fundamentaltransform resolution, using the component frequencies estimated in theCSPE, sampling a set of frequency domain high resolution windows toselect a high resolution window that fits at least one of the amplitude,phase, amplitude modulation and frequency modulation of a component ofan underlying signal wherein the component comprises at least oneoscillator peak and providing an output signal in the form of amathematical representation stored in a computer-accessible form.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain signal comprises receiving a timedomain signal stream and creating a first windowed data set and a secondwindowed data set comprising samples from the signal stream multipliedby an analysis window, wherein an initiation of the second window lagsan initiation of the first window, converting the first and secondwindowed data sets to a frequency domain and storing the resulting dataas frequency domain data having a fundamental transform resolution,performing complex spectral phase evolution (CSPE) on the frequencydomain data to estimate component frequencies of the first and secondwindowed data sets at a resolution greater than the fundamentaltransform resolution, using the component frequencies estimated in theCSPE, sampling a set of frequency domain high resolution windows toselect a high resolution window that fits at least one of the amplitude,phase, amplitude modulation and frequency modulation of a component ofan underlying signal wherein the component comprises at least oneoscillator peak and providing an output signal in the form of a featurevector adapted for use in a speech processing system.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain signal comprises receiving a timedomain signal stream and creating a first windowed data set and a secondwindowed data set comprising samples from the signal stream multipliedby an analysis window, wherein an initiation of the second window lagsan initiation of the first window, converting the first and secondwindowed data sets to a frequency domain and storing the resulting dataas frequency domain data having a fundamental transform resolution,performing complex spectral phase evolution (CSPE) on the frequencydomain data to estimate component frequencies of the first and secondwindowed data sets at a resolution greater than the fundamentaltransform resolution, using the component frequencies estimated in theCSPE, sampling a set of frequency domain high resolution windows toselect a high resolution window that fits at least one of the amplitude,phase, amplitude modulation and frequency modulation of a component ofan underlying signal wherein the component comprises at least oneoscillator peak and reproducing a selected portion of the originalsignal as an output signal.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain signal comprises receiving a timedomain signal stream and creating a first windowed data set and a secondwindowed data set comprising data from the signal stream multiplied byan analysis window, wherein an initiation of the second window lags aninitiation of the first window, converting the first and second windoweddata sets to a frequency domain and storing the resulting data asfrequency domain data having a fundamental transform resolution,performing complex spectral phase evolution (CSPE) on the frequencydomain data to estimate component frequencies of the first and secondwindowed data sets at a resolution greater than the fundamentaltransform resolution, using the component frequencies estimated in theCSPE, sampling a set of frequency domain high resolution windows toselect a high resolution window that fits at least one of the amplitude,phase, amplitude modulation and frequency modulation of a component ofan underlying signal wherein the component comprises at least oneoscillator peak and providing an output signal in the form of amathematical representation stored in a computer-accessible form.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain signal comprises receiving a timedomain signal stream and creating a first windowed data set and a secondwindowed data set comprising samples from the signal stream multipliedby an analysis window, wherein an initiation of the second window lagsan initiation of the first window, converting the first and secondwindow data sets to a frequency domain and storing the resulting data asfrequency domain data having a fundamental transform resolution,performing complex spectral phase evolution (CSPE) on the frequencydomain data to estimate component frequencies of the first and secondwindowed data sets at a resolution greater than the fundamentaltransform resolution, using the component frequencies estimated in theCSPE, sampling a set of frequency domain high resolution windows toselect a high resolution window that fits at least one of the amplitude,phase, amplitude modulation and frequency modulation of a component ofan underlying signal wherein the component comprises at least oneoscillator peak and providing an output signal in the form of a featurevector adapted for use in a speech processing system.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain signal comprises receiving a timedomain signal stream and creating a first window data set and a secondwindow data set comprising samples from the signal stream multiplied byan analysis window, wherein an initiation of the second window lags aninitiation of the first window converting the first and second windoweddata sets to a frequency domain and storing the resulting data asfrequency domain data having a fundamental transform resolutionperforming complex spectral phase evolution (CSPE) on the frequencydomain data to estimate component frequencies of the first and secondwindowed data sets at a resolution greater than the fundamentaltransform resolution, using the component frequencies estimated in theCSPE, sampling a set of frequency domain high resolution windows toselect a high resolution window that fits at least one of the amplitude,phase, amplitude modulation and frequency modulation of a component ofan underlying signal wherein the component comprises at least oneoscillator peak and grouping tracklets that emanate from a single sourcewithin the underlying signal and reproducing a selected portion of theoriginal signal as an output signal.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain signal comprises receiving a timedomain signal stream and creating a first windowed data set and a secondwindowed data set comprising samples from the signal stream multipliedby an analysis window, wherein an initiation of the second window lagsan initiation of the first window, converting the first and secondwindow data sets to a frequency domain and storing the resulting data asfrequency domain data having a fundamental transform resolution,performing complex spectral phase evolution (CSPE) on the frequencydomain data to estimate component frequencies of the first and secondwindowed data sets at a resolution greater than the fundamentaltransform resolution, using the component frequencies estimated in theCSPE, sampling a set of frequency domain high resolution windows toselect a high resolution window that fits at least one of the amplitude,phase, amplitude modulation and frequency modulation of a component ofan underlying signal wherein the component comprises at least oneoscillator peak; and grouping tracklets that emanate from a singlesource within the underlying signal and providing an output signal inthe form of a mathematical representation stored in acomputer-accessible form.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain signal comprises receiving a timedomain signal stream and creating a first windowed data set and a secondwindowed data set comprising samples from the signal stream multipliedby an analysis window, wherein an initiation of the second window lagsan initiation of the first window, converting the first and secondwindowed data sets to a frequency domain and storing the resulting dataas frequency domain data having a fundamental transform resolution,performing complex spectral phase evolution (CSPE) on the frequencydomain data to estimate component frequencies of the first and secondwindowed data sets at a resolution greater than the fundamentaltransform resolution, using the component frequencies estimated in theCSPE, sampling a set of frequency domain high resolution windows toselect a high resolution window that fits at least one of the amplitude,phase, amplitude modulation and frequency modulation of a component ofan underlying signal wherein the component comprises at least oneoscillator peak and grouping tracklets that emanate from a single sourcewithin the underlying signal and providing an output signal in the formof a feature vector adapted for use in a speech processing system.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain signal comprises representing aplurality of channels each comprising a first window comprising a firstset and a second window comprising a second set of frequency domain datain a unified domain representation and performing complex spectral phaseevolution (CSPE) on the frequency domain data to estimate componentfrequencies of the frequency domain data at a resolution greater thanthe fundamental transform resolution of the frequency domain data,including using the phase rotation measured between the frequency domainrepresentation of two time-separated sample windows to detect an actualunderlying frequency at said greater resolution.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain audio signal comprises receiving atime domain signal stream and creating a first windowed data set and asecond windowed data set comprising samples from the signal streammultiplied by an analysis window, wherein an initiation of the secondwindow lags an initiation of the first window, converting the first andsecond windowed data sets to a frequency domain and storing theresulting data as frequency domain data having a fundamental transformresolution, performing complex spectral phase evolution (CSPE) on thefrequency domain data to estimate component frequencies of the first andsecond windowed data sets at a resolution greater than the fundamentaltransform resolution and using the component frequencies estimated inthe CSPE, sampling a set of frequency domain high resolution windows toselect a high resolution window that fits at least one of the amplitude,phase, amplitude modulation and frequency modulation of a component ofan underlying signal wherein the component comprises at least oneoscillator peak.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain video signal comprises receiving atime domain signal stream such as can be created by scanning rows orcolumns of a digital image or video frame and creating a first windoweddata set, and a second windowed data set comprising samples from thesignal stream and optionally multiplied by an analysis window, whereinan initiation of the second window lags an initiation of the firstwindow, converting the first and second windowed data sets to afrequency domain and storing the resulting data as frequency domain datahaving a fundamental transform resolution, performing complex spectralphase evolution (CSPE) on the frequency domain data to estimatecomponent frequencies of the first and second windowed data sets at aresolution greater than the fundamental transform resolution and usingthe component frequencies estimated in the CSPE, sampling a set offrequency domain high resolution windows to select a high resolutionwindow that fits at least one of the amplitude, phase, amplitudemodulation and frequency modulation of a component of an underlyingsignal wherein the component comprises at least one oscillator peak.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain signal comprises receiving a timedomain signal stream and creating a first windowed data set and a secondwindowed data set comprising samples from the signal stream multipliedby an analysis window, wherein an initiation of the second window lagsan initiation of the first window, converting the first and secondwindow data sets to a frequency domain and storing the resulting data asfrequency domain data having a fundamental transform resolution,performing complex spectral phase evolution (CSPE) on the frequencydomain data to estimate component frequencies of the first and secondwindowed data sets at a resolution greater than the fundamentaltransform resolution, using the component frequencies estimated in theCSPE, sampling a set of frequency domain high resolution windows toselect a high resolution window that fits at least one of the amplitude,phase, amplitude modulation and frequency modulation of a component ofan underlying signal wherein the component comprises at least oneoscillator peak and using a tracking algorithm to identify at least onetracklet of oscillator peaks that emanate from a single oscillatorsource within the underlying signal.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain signal comprises receiving a timedomain signal stream and creating a first windowed data set and a secondwindowed data set comprising samples from the signal stream multipliedby an analysis window, wherein an initiation of the second window lagsan initiation of the first window, converting the first and secondwindow data sets to a frequency domain and storing the resulting data asfrequency domain data having a fundamental transform resolution,performing complex spectral phase evolution (CSPE) on the frequencydomain data to estimate component frequencies of the first and secondwindowed data sets at a resolution greater than the fundamentaltransform resolution, using the component frequencies estimated in theCSPE, sampling a set of frequency domain high resolution windows toselect a high resolution window that fits at least one of the amplitude,phase, amplitude modulation and frequency modulation of a component ofan underlying signal wherein the component comprises at least oneoscillator peak and using a tracking algorithm to identify at least onetracklet of oscillator peaks that emanate from a single oscillatorsource within the underlying signal, wherein the tracking algorithm usesinformation from the CSPE to predict the behavior of an oscillatorcomponent of a signal.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain signal comprises receiving a timedomain signal stream and creating a first windowed data set and a secondwindowed data set comprising samples from the signal stream multipliedby an analysis window, wherein an initiation of the second window lagsan initiation of the first window, converting the first and secondwindow data sets to a frequency domain and storing the resulting data asfrequency domain data having a fundamental transform resolution,performing complex spectral phase evolution (CSPE) on the frequencydomain data to estimate component frequencies of the first and secondwindowed data sets at a resolution greater than the fundamentaltransform resolution, using the component frequencies estimated in theCSPE, sampling a set of frequency domain high resolution windows toselect a high resolution window that fits at least one of the amplitude,phase, amplitude modulation and frequency modulation of a component ofan underlying signal wherein the component comprises at least oneoscillator peak, using a tracking algorithm to identify at least onetracklet of oscillator peaks that emanate from a single oscillatorsource within the underlying signal and grouping tracklets that emanatefrom a single source.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain signal comprises receiving a timedomain signal stream and creating a first windowed data set and a secondwindowed data set comprising samples from the signal stream multipliedby an analysis window, wherein an initiation of the second window lagsan initiation of the first window, converting the first and secondwindow data sets to a frequency domain and storing the resulting data asfrequency domain data having a fundamental transform resolution,performing complex spectral phase evolution (CSPE) on the frequencydomain data to estimate component frequencies of the first and secondwindowed data sets at a resolution greater than the fundamentaltransform resolution, using the component frequencies estimated in theCSPE, sampling a set of frequency domain high resolution windows toselect a high resolution window that fits at least one of the amplitude,phase, amplitude modulation and frequency modulation of a component ofan underlying signal wherein the component comprises at least oneoscillator peak, using a tracking algorithm to identify at least onetracklet of oscillator peaks that emanate from a single oscillatorsource within the underlying signal, grouping tracklets that emanatefrom a single source and receiving a plurality of signal streams andcreating first and second sets of input sample windows eachcorresponding to one of the plurality of signal streams, wherein aninitiation of the second set of input samples time lags an initiation ofthe first set of input samples.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain signal comprises receiving a timedomain signal stream and creating a first windowed data set and a secondwindowed data set comprising samples from the signal stream multipliedby an analysis window, wherein an initiation of the second window lagsan initiation of the first window, converting the first and secondwindowed data sets to a frequency domain and storing the resulting dataas frequency domain data having a fundamental transform resolution,performing complex spectral phase evolution (CSPE) on the frequencydomain data to estimate component frequencies of the first and secondwindowed data sets at a resolution greater than the fundamentaltransform resolution, using the component frequencies estimated in theCSPE, sampling a set of frequency domain high resolution windows toselect a high resolution window that fits at least one of the amplitude,phase, amplitude modulation and frequency modulation of a component ofan underlying signal wherein the component comprises at least oneoscillator peak, using a tracking algorithm to identify at least onetracklet of oscillator peaks that emanates from a single oscillatorsource within the underlying signal, grouping tracklets that emanatefrom a single source and providing an output signal.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain signal comprises performing complexspectral phase evolution (CSPE) on frequency domain data to estimatecomponent frequencies of the frequency domain data at a resolutiongreater than a fundamental transform resolution of the frequency domaindata and using the component frequencies estimated in the CSPE, samplinga set of stored high resolution windows to select a high resolutionwindow that fits at least one of the amplitude, phase, amplitudemodulation and frequency modulation of the underlying signal component.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain signal comprises performing complexspectral phase evolution (CSPE) on frequency domain data to estimatecomponent frequencies of the frequency domain data at a resolutiongreater than a fundamental transform resolution of the frequency domaindata and determining an estimated frequency modulation in one or moreoscillator peaks in a windowed data set, applying a plurality offrequency modulation pullback operators (FMPO) to the sample data,applying at least one of a non-linear interpolation, a linearinterpolation, and an extrapolation to the resulting data to provide animproved estimate of the frequency modulation of the underlying signalcomponent, and storing the improved estimate in a further modified highresolution window data set.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain signal comprises receiving a timedomain signal stream and creating a first windowed data set and a secondwindowed data set comprising samples from the signal stream multipliedby an analysis window, wherein an initiation of the second window lagsan initiation of the first window, converting the first and secondwindowed data sets to a frequency domain and storing the resulting dataas frequency domain data having a fundamental transform resolution,performing complex spectral phase evolution (CSPE) on the frequencydomain data to estimate component frequencies of the first and secondwindowed data sets at a resolution greater than the fundamentaltransform resolution, using the component frequencies estimated in theCSPE, sampling a set of frequency domain high resolution windows toselect a high resolution window that fits at least one of the amplitude,phase, amplitude modulation and frequency modulation of a component ofan underlying signal wherein the component comprises at least oneoscillator peak and separating the underlying signal into a plurality ofsignal components each corresponding to one of a plurality of distinctsources, reconstructing a single merged signal that best represents theplurality of signal components, and providing the merged signal as anoutput.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain signal comprises receiving a timedomain signal stream and creating a first windowed data set and a secondwindowed data set comprising samples from the signal stream multipliedby an analysis window, wherein an initiation of the second window lagsan initiation of the first window, converting the first and secondwindowed data sets to a frequency domain and storing the resulting dataas frequency domain data having a fundamental transform resolution,performing complex spectral phase evolution (CSPE) on the frequencydomain data to estimate component frequencies of the first and secondwindowed data sets at a resolution greater than the fundamentaltransform resolution, using the component frequencies estimated in theCSPE, sampling a set of frequency domain high resolution windows toselect a high resolution window that fits at least one of the amplitude,phase, amplitude modulation and frequency modulation of a component ofan underlying signal wherein the component comprises at least oneoscillator peak, using a tracking algorithm to identify at least onetracklet of oscillator peaks that emanate from a single oscillatorsource within the underlying signal and separating the underlying signalinto a plurality of signal components each corresponding to one of aplurality of distinct sources, reconstructing a single merged signalthat best represents the plurality of signal components, and providingthe merged signal as an output.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain signal comprises receiving a timedomain signal stream and creating a first windowed data set and a secondwindowed data set comprising samples from the signal stream, wherein aninitiation of the second window lags an initiation of the first window,converting the first and second windowed data sets to a frequency domainand storing the resulting data as frequency domain data having afundamental transform resolution, performing complex spectral phaseevolution (CSPE) on the frequency domain data to estimate componentfrequencies of the first and second windowed data sets at a resolutiongreater than the fundamental transform resolution, using the componentfrequencies estimated in the CSPE, sampling a set of frequency domainhigh resolution windows to select a high resolution window that fits atleast one of the amplitude, phase, amplitude modulation and frequencymodulation of a component of an underlying signal wherein the componentcomprises at least one oscillator peak, using a tracking algorithm toidentify at least one tracklet of oscillator peaks that emanate from asingle oscillator source within the underlying signal, groupingtracklets that emanate from a single source and separating theunderlying signal into a plurality of signal components eachcorresponding to one of a plurality of distinct sources, reconstructinga single merged signal that best preserves desired features, andproviding the merged signal as an output.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain signal comprises receiving a timedomain signal stream and creating a first windowed data set and a secondwindowed data set comprising samples from the signal stream multipliedby an analysis window, wherein an initiation of the second window lagsan initiation of the first window, converting the first and secondwindows to a frequency domain and storing the resulting data asfrequency domain data having a fundamental transform resolution,performing complex spectral phase evolution (CSPE) on the frequencydomain data to estimate component frequencies of the frequency domaindata at a resolution greater than the fundamental transform resolutionof the frequency domain data, using the component frequencies estimatedin the CSPE, sampling a set of stored high resolution windows in asinglet transformation process to select a high resolution window thatfits the amplitude, phase, amplitude modulation and frequency modulationof the underlying signal component comprising at least one oscillatorpeak and removing the effects of the estimated component from at leastone of the stored windowed data sets, using a tracking algorithm toidentify at least one tracklet of oscillator peaks that emanate from asingle oscillator source within the underlying signal, wherein thetracking algorithm uses information from the CSPE to predict thebehavior of an oscillator component of a signal, grouping tracklets thatemanate from a single source, wherein grouping is aided by a visualrepresentation of a plurality of tracklets displayed in a graphical userinterface that enables at least one of selection, deletion andassociation of a tracklet and providing an output signal whereinconverting the first and second sets of input samples comprisesconverting the first and second sets of input samples to the frequencydomain using at least one of a Discrete Fourier transform (DFT) and aFast Fourier Transform (FFT), and any related transform.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain signal comprises receiving aplurality of signal streams and creating first and second sets of inputsample windows each corresponding to one of the plurality of signalstreams, wherein an initiation of the second set of input samples timelags an initiation of the first set of input samples and whereinconverting the first and second sets of input samples comprisesconverting the first and second sets of input samples to the frequencydomain using at least one of a Discrete Fourier transform (DFT) and aFast Fourier Transform (FFT), further comprising multiplying the firstset of input samples and the second set of input samples with ananalysis window, converting the first and second input sample windows toa frequency domain, modifying the converted window by adding at leastone of an amplitude effect and a frequency effect, and storing theresulting modified window data set, representing a plurality of channelseach comprising a first set and a second set of frequency domain data ina unified domain representation and performing complex spectral phaseevolution (CSPE) on the frequency domain data to estimate componentfrequencies of the frequency domain data at a resolution greater thanthe fundamental transform resolution of the frequency domain data,including using the phase rotation measured between two time-separatedsample windows to detect an actual underlying frequency at said greaterresolution, using the component frequencies estimated in the CSPE,sampling a set of stored high resolution windows in a singlettransformation process to select a high resolution window that fits theamplitude, phase, amplitude modulation and frequency modulation of theunderlying signal component and removing the effects of the estimatedcomponent from at least one of the stored windowed data sets, using atracking algorithm to identify at least one tracklet of oscillator peaksthat emanate from a single oscillator source within the underlyingsignal, wherein the tracking algorithm uses information from the CSPE topredict the behavior of an oscillator component of a signal, groupingtracklets that emanate from a single source, wherein grouping is aidedby a visual representation of a plurality of tracklets displayed in agraphical user interface that enables at least one of selection,deletion and association of a tracklet and providing an output signal.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain signal comprises receiving a timedomain signal stream and creating a first set of input samples in thetime domain and a second set of input samples in the time domain fromthe signal stream, wherein an initiation of the second set of inputsamples time lags an initiation of the first set of input samples,multiplying the sets of input samples by an analysis window, convertingthe first and second input sample windows to a frequency domain andstoring the resulting transformed windowed data set for analysis,performing complex spectral phase evolution (CSPE) on the frequencydomain data to estimate component frequencies of the frequency domaindata at a resolution greater than the fundamental transform resolutionof the frequency domain data, using the component frequencies estimatedin the CSPE, sampling a set of stored high resolution windows to selecta high resolution window that fits at least one of the amplitude, phase,amplitude modulation and frequency modulation of the underlying signalcomponent comprising at least one oscillator peak, determining anestimated frequency modulation in a stored high resolution window dataset, applying a plurality of frequency modulation pullback operators(FMPO) to the sample data, applying at least one of a non-linearinterpolation, a linear interpolation, and an extrapolation to theresulting data to provide an improved estimate of the frequencymodulation of the underlying signal component, and storing the improvedestimate in a further modified high resolution window data set, using atracking algorithm to identify at least one tracklet of oscillator peaksthat emanate from a single oscillator source within the underlyingsignal, grouping tracklets that emanate from a single source andproviding an output signal.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a time domain signal comprises receiving aplurality of signal streams and creating first and second sets of inputsample windows each corresponding to one of the plurality of signalstreams, wherein an initiation of the second set of input samples timelags an initiation of the first set of input samples and whereinconverting the first and second sets of input samples comprisesconverting the first and second sets of input samples to the frequencydomain using at least one of a Discrete Fourier transform (DFT) and aFast Fourier Transform (FFT) or a discrete cosine transform or anotherrelated transform, further comprising multiplying the first set of inputsamples and the second set of input samples with an analysis window,converting the first and second input sample windows to a frequencydomain, modifying the converted window by adding at least one of anamplitude effect and a frequency effect, and storing the resultingmodified windowed data set, representing a plurality of channels eachcomprising a first set and a second set of frequency domain data in aunified domain representation and performing complex spectral phaseevolution (CSPE) on the frequency domain data to estimate componentfrequencies of the frequency domain data at a resolution greater thanthe fundamental transform resolution of the frequency domain data,including using the phase rotation measured between two time-separatedsample windows to detect an actual underlying frequency at said greaterresolution, using the component frequencies estimated in the CSPE,sampling a set of stored high resolution windows in a singlettransformation process to select a high resolution window that fits atleast one of the amplitude, phase, amplitude modulation and frequencymodulation of the underlying signal component and removing the effectsof the estimated component from at least one of the stored windowed datasets, using a tracking algorithm to identify at least one tracklet ofoscillator peaks that emanate from a single oscillator source within theunderlying signal, wherein the tracking algorithm uses information fromthe CSPE to predict the behavior of an oscillator component of a signal,grouping tracklets that emanate from a single source, wherein groupingis aided by a visual representation of a plurality of trackletsdisplayed in a graphical user interface that enables at least one ofselection, deletion and association of a tracklet and separating theunderlying signal into a plurality of signal components eachcorresponding to one of a plurality of distinct sources, and optionallyreconstructing a single merged signal that best represents the pluralityof signal components, and providing the (optionally) merged signal as anoutput.

In accordance with another exemplary and non-limiting embodiment, amethod of processing a signal comprises receiving a plurality of signalstreams each comprising a substantial amount of ambient noise orinterfering signals and creating first and second sets of input samplewindows each corresponding to one of the plurality of signal streams,wherein an initiation of the second set of input samples time lags aninitiation of the first set of input samples, multiplying the first andsecond sample windows by an analysis window, converting the first andsecond input sample windows to a frequency domain and storing theresulting data, performing complex spectral phase evolution (CSPE) onthe frequency-domain data to estimate component frequencies of the dataset at a resolution greater than the fundamental transform resolution,using the component frequencies estimated in the CSPE, sampling a set ofstored high resolution windows to select a high resolution window thatfits at least one of the amplitude, phase, amplitude modulation andfrequency modulation of the underlying signal component, using atracking algorithm to identify at least one tracklet of oscillator peaksthat emanate from a single oscillator source within the underlyingsignal, grouping tracklets that emanate from a single source, rejectingtracklets that are likely to be associated with noise or interferingsignals, selecting at least one grouping of tracklets, reconstructing asignal from the selected groupings of tracklets and providing the signalas an output.

BRIEF DESCRIPTION OF THE FIGURES

In the figures, which are not necessarily drawn to scale, like numeralsmay describe substantially similar components throughout the severalviews. Like numerals having different letter suffixes may representdifferent instances of substantially similar components. The figuresillustrate generally, by way of example, but not by way of limitation,certain embodiments discussed in the present document.

FIG. 1 is an illustration of a signal extraction process according to anexemplary and non-limiting embodiment;

FIG. 2 illustrates signal extraction processing steps according to anexemplary and non-limiting embodiment;

FIG. 3 illustrates a method for pre-processing the source signal using asingle channel pre-processor according to an exemplary and non-limitingembodiment;

FIG. 4 illustrates a method for pre-processing the source signal usingthe single channel pre-processor to detect frequency modulation withinthe signal according to an exemplary and non-limiting embodiment;

FIG. 5 illustrates a single channel super-resolution algorithm accordingto an exemplary and non-limiting embodiment;

FIG. 6 illustrates a method for generating high accuracy frequency andAM and FM modulation estimates such as to enable the extraction of a setof signal components according to an exemplary and non-limitingembodiment;

FIG. 7 illustrates an example of a method for unified domain superresolution according to an exemplary and non-limiting embodiment;

FIG. 8 illustrates an example of a method for unified domain superresolution with amplitude and frequency modulation detection accordingto an exemplary and non-limiting embodiment;

FIG. 9 illustrates a graphical representation of FFT spectrum accordingto an exemplary and non-limiting embodiment;

FIG. 10 illustrates an example of a method for creating high-resolutionwindows for AM/FM detection according to an exemplary and non-limitingembodiment;

FIG. 11 illustrates an example of a method for frequency modulationdetection according to an exemplary and non-limiting embodiment;

FIG. 12 illustrates a modulation detection decision tree according to anexemplary and non-limiting embodiment;

FIG. 13 illustrates an example of a method performed by a signalcomponent tracker according to an exemplary and non-limiting embodiment;

FIG. 14 illustrates an example of a method performed by the signalcomponent tracker that may use frequency and phase prediction accordingto an exemplary and non-limiting embodiment;

FIG. 15 is an illustration of a computer generated interface for tabletor cell phone control according to an exemplary and non-limitingembodiment;

FIG. 16 is an illustration of a track editor according to an exemplaryand non-limiting embodiment;

FIG. 17 is an illustration of a track editor sub-selection according toan exemplary and non-limiting embodiment; and

FIG. 18 is an illustration of track editor data visualizer according toan exemplary and non-limiting embodiment.

DETAILED DESCRIPTION

FIG. 1 illustrates an exemplary and non-limiting embodiment of a method100 for source signal separation. In an example, a representative inputsignal may be a source signal (SS) including an audio signal/sound as aninput to the system such that the SS is a source agnostic and may beused with respect to any type of source signal. Other representativeinput signals may include but are not limited to ambient sound, audio,video, speech, image, communication, geophysical, SONAR, RADAR, thermal,optical/light, medical, and musical signals. The method 100 may includeone or more steps that may be used in combination or in part to analyzethe SS, separate the SS into its constituent elements, and thenreconstitute the SS signal in whole or in part.

As shown in FIG. 1, the method 100 may be configured to select a signalat step 102 so as to process the signal for the signal separation. In anexample, contiguous samples (referred to herein as “windows” or “samplewindows” that may represent windows of samples in time) may be selectedfor analysis. Typically, multiple windows may be selected with a smalltime-delay between them. Further, at step 104, the method 100 may beconfigured to multiply the SS (i.e., in the form of contiguous samples)with an analysis window such as a window B1 as illustrated in FIG. 1.The analysis window may also be referred to herein as a taper.

At step 108, a high resolution window (HRW) such as a HRW C1 may becreated. Further, a copy of the analysis window used for signalpreparation may be converted to a high-resolution frequency domain andstored for oscillator peak analysis. Optionally, sets of HRWs may bestored that have amplitude and frequency modulation effects addedtherein. At step 110, a conversion to Frequency Domain and ComplexSpectral Phase Evolution (CSPE) high-resolution frequency estimate maybe performed. In an example, time-domain windows are converted to thefrequency domain via a transform, such as a Fast Fourier Transform(FFT), the Discrete Fourier Transform (DFT) the Discrete CosineTransform (DCT) or other related transform. The accuracy of frequencyestimates created by such transforms may be conventionally limited bythe number of input samples. The CSPE transform overcomes theselimitations and provides a set of highly accurate frequency estimates.In particular, the CSPE calculation uses the phase rotation measuredbetween the transforms of two time-separated sample windows to detectthe actual underlying frequency.

At step 112, the method 100 may be configured to identify oscillatorpeak parameters via a Singlet Transform Process. Specifically, highresolution windows (HRWs) are sampled to select the HRW with the mostaccurate fit to estimate the amplitude, phase, amplitude modulation andfrequency modulation of the underlying signal component using highaccuracy frequency estimates that are provided by the CSPE calculation.In some embodiments, one may remove the effects of this component sothat estimates of nearby oscillators may become more accurate. Thesinglet transform process may be reversed to re-produce portions of orthe entire original frequency domain signal. At step 114, the method 100may be configured to perform tracking and grouping. In an example, thetracking may be performed to identify oscillator peaks that may emanatefrom a single oscillator using tracking algorithms, such as a singleharmonic produced by a musical instrument or a person's voice. A set ofoscillator peaks that has been determined to be emanating from a singlesource is called a tracklet. In an example, the grouping may beperformed to identify tracklets that emanate from a single source. Forexample, such a grouping can include multiple harmonics of a singlemusical instrument or person's voice. A set of tracklets that has beendetermined to be emanating from a single source is called a coherentgroup.

At step 118, the oscillator peaks may be output at any stage after thesinglet transform process. Further, the information gathered in thetracking and grouping stages may be used to select a set of desiredoscillator peaks. In an example, some or all oscillator peaks may beconverted accurately into some or all of the original signal formatsusing the singlet transform process. In another example, some or alloscillator peaks may be converted into another format, such as a featurevector that may be used as an input to a speech recognition system ormay be further transformed through a mathematical function directly intoa different output format. The above steps may be used to analyze,separate and reconstitute any type of signal. The output of this systemmay be in the same form as the original signal or may be in the form ofa mathematical representation of the original signal for subsequentanalysis.

As used herein in the detailed description, a “frequency-phaseprediction” is a method for predicting the frequency and phase evolutionof a tracklet composed of oscillator peaks. As used herein, a “featurevector” is a set of data that has been measured from a signal. Inaddition, commonly feature vectors are used as the input to speechrecognition systems. As used herein, “Windowed transform” refers topre-multiplying an original sample window by a “taper” or windowingfunction (e.g., Hanning, Hamming, boxcar, triangle, Bartlett, Blackman,Chebyshev, Gaussian and the like) to shape spectral peaks differently.As used herein, “Short” refers, generally, to a finite number of samplesthat is appropriate to a given context and may include several thousandor several hundreds of samples, depending on the sample rate, such as ina Short Time Fourier Transform (STFT). For example, an audio CD includes44100 samples per second, so a short window of 2048 samples is stillonly about 1/20th of a second. As used herein a “tracklet” refers to aset of oscillator peaks from different frames that a tracker hasdetermined to be from the same oscillator. As used herein, a“Mahalanobis Distance” refers to a well-known algorithm in the art formeasuring the distance between two multi-dimensional points that takesuncertainty measures into account. This algorithm is commonly used intracking applications to determine the likelihood that a tracklet and ameasurement should be combined or assigned to the same source or sametracklet. As used herein, “tracklet association” refers to a method fordetermining which new measurements should be combined with whichexisting tracklets. As used herein, “greedy association” refers to analgorithm known in the art for performing tracklet association. As usedherein, “partitioning” refers to a method for dividing tracklets intodistinct groups. Generally these groups will correspond to distinctsound emitters, such as a person speaking. As used herein, a “unionfind” is an algorithm known in the art for partitioning. As used herein,a “coherent group” refers to a set of tracklets that have beendetermined to be from the same signal emitter, such as a personspeaking. As used herein, a “Mel Frequency Complex Coefficient” is awell-known type of feature commonly used as the input to speechrecognition systems.

In accordance with one or more embodiments, the methods and systems forSS disclosed herein may facilitate separation of a source signal into aplurality of signal elements. The methods and systems described hereinmay be used in whole or in part to isolate and enhance individualelements in the source signal. The systems and methods may be applied togenerally any signal source to achieve signal separation.

In accordance with one or more embodiments, the methods and systems forSS may facilitate execution of a series of algorithms that may be usedin part or in combination to perform signal separation and enhancement.The series of algorithms may be implemented in hardware, software, or acombination of hardware and software.

In accordance with one or more embodiments, the methods and systems forSS may be configured to a pre-processor that may be a single-channel ora multi-channel, and a super-resolution module that may be asingle-channel or a multi-channel. In accordance with one or moreembodiments, the methods for SS may include a family of methods that maybe based on Complex Spectral Phase Evolution, including methods forshort-time stable sinusoidal oscillations, short-time linear frequencymodulation methods, time-varying amplitude modulation methods, jointamplitude and frequency modulation methods, and a Singlet Representationmethod. As used herein, FM-CSPE refers to the specific methods withinthe family of CSPE methods that apply to frequency modulating signals.Similarly, AM-CSPE refers to the specific methods within the family ofCSPE methods that apply to amplitude modulating signals.

The methods and systems for SS described herein can provide one or moreof the following advantages. For example, the methods and systems mayfacilitate extraction of interfering elements from the source signalseparately and unwanted elements may be removed from the source signal.In an example, targeted elements of the source signal may be extractedor isolated without corrupting the targeted element using the methodsand systems for SS. In another example, overlapping signal elementswithin the same frequency range may be independently extracted andenhanced despite the convolution effects of the measurement process(also known as “smearing” or the “uncertainty principle”). The methodsand systems for SS as described herein may facilitate provisioning of adetailed analysis of the source signal due to an increase in an accuracyof the processing techniques of the methods and systems for SS disclosedherein with respect to current processing techniques.

In accordance with one or more embodiments, the methods and systems forSS may be configured to include a signal component tracker that may beconfigured to implement a method for grouping signal components in time,and/or by harmonics, and/or by other similarity characteristics toidentify coherent sources. In accordance with one or more embodiments,the methods and systems for SS may be configured to include a coherentstructure aggregator and a coherent structure selector/separator suchthat the coherent structure selector/separator may be configured toimplement a method for identifying coherent structures for extraction,isolation, enhancement, and/or re-synthesis. In accordance with one ormore embodiments, the methods and systems may be configured to include aunified domain transformation and unified domain complex spectral phaseevolution (CSPE) such as to combine multiple signal channels into asingle mathematical structure and to utilize a version of the CSPEmethods designed to work in the unified domain. The methods and systemsfor SS may be configured to include a re-synthesis module that mayfacilitate generation of a frequency domain signal from a set ofoscillator peaks. The re-synthesis module may be implemented using asingle-channel or a multi-channel module.

In accordance with one or more embodiments, the SS system may beconfigured to include a multi-channel preprocessor, a multi-channelsuper-resolution module, a tracker/aggregator/selector/separator, and amulti-channel re-synthesis module. In accordance with one or moreembodiments, the methods for SS may be configured to include one or moreof the operations such as a complex spectral phase evolution (CSPE), asinglet representation method, a unified domain transformation, aunified domain complex spectral phase evolution, a signal componenttracking, a coherent structure aggregation, a coherent structureseparation, a coherent structure reconstruction in the time domain, anambient signal remixing or reconstitution and other operations.

The CSPE operation may refer to a method for overcoming the accuracylimitations of the Fast Fourier Transform (FFT) or Discrete FourierTransform (DFT). The CSPE operation may improve an accuracy of FFT-basedspectral processing, in some embodiments from 21.5 Hz to the order of0.1 Hz. In some embodiments, the accuracy may be better than 0.1 Hz. Inaccordance with one or more embodiments, the CSPE operations may beconfigured to include short-time stable sinusoidal oscillation methods,short-time linear frequency modulation methods, time-varying amplitudemodulation methods, and joint amplitude and frequency modulationmethods.

The singlet representation method refers to a method by which ashort-time stable or quasi-stable oscillator may be projected into afrequency domain signal or extracted from a frequency domain signal. Inan example, the oscillator may refer to any source of oscillation,including but not limited to a sinusoidal oscillation, a short-timestable oscillation of any duration, a quasi-stable oscillation, or asignal that may be created to a desired degree of accuracy by a finitesum of such oscillators. The singlet transformation or singletrepresentation may include information on an amplitude, phase and(super-resolution) frequency of the oscillator, along with informationabout the smearing characteristics of the oscillator that may indicatethe degree of interference with other signal elements. Further, thesinglet representation can include information about the smearing andinterference characteristics as a function of the number of decibels ofinterference in a given frequency bin of the original FFT or DFT. Insome embodiments, the singlet representation may include informationabout the (super-resolution) frequency modulation, amplitude modulationand joint frequency-amplitude modulation characteristics.

The unified domain transformation may refer to a method for combiningmultiple signal channels into a single mathematical structure and theunified domain complex spectral phase evolution may refer to a versionof the CSPE methods designed to work in the Unified Domain. The signalcomponent tracking may refer to a method for grouping signal componentsin time, and/or by harmonics, and/or by other similarity characteristicsto identify coherent sources. The coherent Structure Separation mayrefer to a method for identifying coherent structures for extraction,isolation, enhancement, and/or re-synthesis and the coherent structurereconstruction may refer to a method for creating a frequency domain ortime domain signal that is composed of selected oscillator peaks. Theambient signal remixing or reconstitution may refer to a method foradding the original signal (or an amplified or attenuated version of theoriginal signal) to the signal created by coherent structurereconstruction in the time domain to generate a signal having certaindesirable characteristics. In an example, an output may include coherentstructure reconstruction in the time domain, an ambient signal remixingor reconstitution, feature vector creation and automatic translationfrom mathematical representation to other output formats.

FIG. 2 illustrates an embodiment of a SS system 200 that may beconfigured to separate the source signal 202 into the plurality ofelements. In accordance with one or more embodiments, the SS system 200may be configured to include one or more components such as a singlechannel pre-processor 204, a single channel super-resolution module 208,a multi-channel pre-processor 210, multi-channel super-resolution module212, tracker/aggregator/selector/separator 214, single channelre-synthesis module 220, and a multi-channel re-synthesis module 222.These components may be implemented in hardware, software, orprogrammable hardware such as a Field Programmable Gate Array (FPGA).

The single channel pre-processor 204 may facilitate in pre-processing(e.g., preparation) of a single-channel time domain signal that may beprocessed by the single channel super-resolution module. The singlechannel super-resolution module 208 may facilitate in detection of a setof oscillator peaks in a signal that has been prepared by the singlechannel pre-processor. The multi-channel pre-processor 210 mayfacilitate in pre-processing (e.g., preparation) of a multi-channel timedomain signal that may be processed by the multi-channelsuper-resolution module 212. The multi-channel super-resolution module212 may facilitate in detection of a set of oscillator peaks in signalthat has been prepared by the multi-channel pre-processor. In one ormore embodiments, the single channel or the multi-channel pre-processorsmay be combined such as to operate as a single component of the system.

The tracker/aggregator/selector/separator (“TASS”) 214 may be configuredto group, separate, and/or select the subset of oscillator peaks. Thesingle channel re-synthesis module 220 may be configured to produce afrequency domain signal from the set of oscillator peaks. Themulti-channel re-synthesis module 222 may be configured to produce amulti-channel frequency domain signal from the set of oscillator peaks,including any number of channels. In one or more embodiments, there-synthesis may be described as being produced by the single channelmodule or the multi-channel module, but these may be combined such as tooperate as a single component of the system.

In accordance with one or more embodiments, the system 200 may beconfigured to utilize or include varying forms of algorithms,implemented in hardware, software or a combination thereof, customizedfor specific applications including but not limited to audio, video,photographic, medical imaging, cellular, communications, radar, sonar,and seismic signal processing systems. As illustrated in FIG. 2, asignal 202 may be received. The signal 202 may include data associatedwith a live-feed such as ambient sound, or prerecorded data, such as arecording of a noisy environment. The received signal 202 may becategorized as a single channel signal or a multi-channel signal. If thesignal 202 has a single channel of data, such as a mono audio signal,the data associated with the signal 202 may be converted to thefrequency domain with the single channel pre-processor 204. Further, oneor more oscillator peaks may be identified in the frequency domainsignal using the single channel super resolution module 208.

Conversely, the signal 202 may be converted to the frequency domainusing the multi-channel processor 210 if the signal has multiplechannels of data, such as a stereo audio signal. Further, the frequencydomain signal may be communicated to the unified domain super resolutionmodule 212 where a unified domain transformation of the frequency datamay be performed and (super-resolution) oscillator peaks in the unifieddomain frequency data may be identified.

In accordance with one or more embodiments, TASS module 214 may beutilized to identify discrete signal sources by grouping peaks and toaggregate oscillator peaks to isolate desired discrete sources. The TASSmodule 214 may be configured to select one or more coherent groups fromthe aggregated oscillator peaks. Accordingly, the one or more coherentgroups of peaks may be separated and delivered as an output in one ormore formats to one or more channels.

In accordance with one or more embodiments, an output signal may bere-synthesized using the components as illustrated in FIG. 2. As anexample and not as a limitation, the oscillator peaks may be convertedto a re-synthesized signal 218 using the single channel re-synthesismodule 220 if the source signal 202 is an originally single-channelsignal. The re-synthesized signal 218 may also be referred herein to asa single channel signal generated using the single channel re-synthesismodule 220. Similarly, the oscillator peaks may be converted to generatethe re-synthesized signal 218 using the multi-channel re-synthesismodule 222 if the source signal 202 is an originally multi-channelsignal. The re-synthesized signal 218 may also be referred herein to asa multi-channel signal when generated using the multi-channelre-synthesis module 222. As illustrated, signal information may beoutputted in the compact form of the analysis parameters; and/or thesignal may be outputted directly into another format, such as one thatcan be achieved by a mathematical transformation from, orreinterpretation of, the analysis parameters. In other embodiments, thesignal information may be outputted as feature vectors that may bepassed directly to another application, such as a speech recognizer or aspeaker identification system.

In accordance with one or more embodiments, the single channelpre-processor 204 may be configured to facilitate preparation of singlechannel time domain signal data for processing by the Single ChannelCSPE super resolution techniques using the single channel superresolution module 208. The input to the single channel pre-processor 204is a single-channel time-domain signal that may be a live feed or arecorded file. In an example, a multi-channel data streams are processedby the multi-channel pre-processor 210 that may be configured to processat least more than one channels of the multi-channel data stream.

Conventional signal analysis systems generally use the DFT or FFT or theDiscrete Cosine Transform (DCT) or related transform to converttime-domain signal data to the frequency-domain for signal analysis andenhancement. The techniques employed in the methods and systems for SSas disclosed herein may be configured to facilitate pre-processing ofthe signal 202 using two (or more) FFTs as building blocks, where thetime-domain input to the second (or more) FFT is a set of samples thatare time delayed with respect to the input to the first FFT.

FIG. 3 illustrates an example embodiment of a method 300 forpre-processing the signal 202 using the single channel pre-processor204. As illustrated, at step 302, the time domain signal stream may bereceived by the single channel pre-processor 204. At step 304, a samplewindow may be filled with n sequential samples of an input signal suchas the signal 202. At step 308, two sampled windows such as a samplewindow A and a sample window B may be created. In an example, a size ofthe sample window A and a number of samples in the sample window A mayoverlap with subsequent and previous sample windows that may bespecified by the user in a parameter file, or may be set as part of thesoftware or hardware implementation. In an example, the sample window Bmay be referred herein to as a time-delayed sample window such that thesample windows A and B may offset in time and the sample window B maylag with sample window A.

At step 310, an analysis window (referred to herein as a taper) may beapplied to the sample window A and sample window B such as to create atapered sample window A and a tapered sample window B respectively. Inan example, the analysis window may be applied using a Hadamard product,whereby two vectors are multiplied together pair wise in a term-by-termfashion. The Hadamard/Schur product is a mathematical operation that maybe defined on vectors, matrices, or generally, arrays. When two suchobjects may have the same shape (and hence the same number of elementsin the same positions), then the Hadamard/Schur product is defined asthe element-by-element product of corresponding entries in the vectors,matrices, or arrays, respectively. This operation is defined, forinstance, in a Matlab programming language to be the operator designatedby “.*”, and in the text below it will be represented either as “.*” oras the operator “⊙” in equations below. As an example, if two vectorsare defined as v₁=[a,b,c,d] and v₂=[e,f,g,h], then the Hadamard/Schurproduct would be the vector v₁⊙v₂=[ae,bf,cg,dh]. In another example, theanalysis window may be chosen to be a standard windowing function suchas the Hanning window, the Hamming window, Welch window, Blackmanwindow, Bartlett window, Rectangular/Boxcar window, or other standardwindowing functions, or other similar analysis window of unique design.At step 312, the tapered sample windows A and B may be converted to afrequency domain using a DFT or FFT or the Discrete Cosine Transform(DCT) or related transform. As a result, FDAT (A) and FDAT (B) may begenerated on conversion such that the FDAT (A) and FDAT (B) are in acomplex form.

FIG. 4 illustrates an example embodiment of a method 400 forpre-processing the signal 202 using the single channel pre-processor 204when frequency modulation detection is required. As illustrated, at step402, the time domain signal stream may be received by the single channelpre-processor 204. At step 404, a sample window may be filled with nsequential samples of an input signal such as the signal 202. At step408, four sampled windows such as a sample window A, a sample window B,a sample window (B_up) and a sample window (B_down) may be created. Inan example, the sample window (B_up) and the sample window (B_down) mayinclude the same samples as the (B) window, but may be processeddifferently. In an example, a size of the sample window A and a numberof samples in the sample window A may overlap with subsequent andprevious sample windows that may be specified by the user in a parameterfile, or may be set as part of the software or hardware implementation.In an example, the sample window B may be referred herein to as atime-delayed sample window such that the sample windows A and B mayoffset in time and the sample window B may lag with sample window A.

At step 410, an analysis window (referred to herein as a taper) may beapplied to the sample window A and sample window B such as to create atapered sample window A and a tapered sample window B respectively. Atstep 412, a modulation pullback operator may be applied to the samplewindow (B_up) and sample window (B_down) such as to create the taperedwindows that can accomplish frequency modulation detection in the signal202. In an example, the frequency modulation detection in the signal 202may be accomplished via the Hadamard product between the sampledmodulation pullback operator and the other samples such as the samplewindow (B_up) and sample window (B_down). For example, a sample window(B_up) may be used with the modulation pullback operator for detectionof positive frequency modulation and a sample window (B_down) may beused with the modulation pullback operator for detection of negativefrequency modulation. At step 414, all four tapered sample windows maybe converted to a frequency domain using a DFT or FFT. As a result, FDAT(A), FDAT(B), FDAT(B_up) and FDAT(B_down) are created in a form ofcomplex spectrum.

The aforementioned methods (e.g., methods 300 and 400) may furtherinclude analyzing an evolution of the complex spectrum from FDAT (A) toFDAT (B) and determining a local phase evolution of the complex spectrumnear each peak in the complex spectrum. The resulting phase change maybe used to determine, on a super-resolved scale that is finer than thatof the FFT or DFT, an underlying frequency that produced the observedcomplex spectral phase evolution. The underlying frequency calculationis an example of super-resolution available through the CSPE method.Further, the method 400 can include analyzing the evolution of thecomplex spectrum from FDAT(A) to FDAT(B_down) and from FDAT(A) toFDAT(B_up) to detect the properties of down modulation and up modulationsuch as to detect presence of the frequency modulation in the signal202.

The methods can further include testing the complex spectral phaseevolution behavior of nearby points in the complex spectrum for each ofthe detected underlying frequencies. The testing may facilitate indetermining whether the behavior of nearby points in the complexspectrum is consistent with the observed behavior near the peaks in thecomplex spectrum. Such approach may be applied to retain well-behavedpeaks and reject inconsistent peaks. Similarly, for each individualmodulating underlying frequency, the methods can include testing thecomplex spectral phase evolution behavior of nearby points in thecomplex spectrum to determine if they evolve in a manner that isconsistent with the observed modulation behavior near the peaks.

The methods can further include conducting a deconvolution analysis todetermine the amplitude and phase of the underlying signal componentthat produced the measured FFT or DFT complex spectrum for eachconsistent peak. Further, a reference frequency, amplitude, phase, andmodulation rate for each consistent modulating peak of the underlyingsignal component that produced the measured FFT or DFT complex spectrummay be determined. The reference frequency is generally set to be at thebeginning or at the center of a frame of time domain samples.

The aforementioned methods as implemented by the single channelpre-processor 204 creates at least two frequency domain data sets thatcan then be processed by single channel CSPE super resolution methods.As discussed, the time domain input to the second set lags the timedomain input to the first set by a small number of samples,corresponding to a slight time delay. Each input is multiplied by theanalysis window and is then transformed to the frequency domain by theDFT or FFT. The frequency domain output of the pre-processor willhenceforth be referred to as FDAT (A) and FDAT (B). In addition, twoadditional frequency domain data sets such as FDAT (B_up) and FDAT(B_down) may be created if frequency modulation detection is required.FDAT (B_up) and FDAT (B_down) are frequency domain representations ofthe time delayed samples contained in the sample window (B) on which themodulation pullback operator is applied before conversion to thefrequency domain. FDAT (B_up) has had a positive frequency modulationpullback operator applied, and FDAT (B_down) has had a negativefrequency modulation pullback operator applied.

Thus, via the inputs, methods and outputs noted above, in accordancewith an exemplary and non-limiting embodiment, a preprocessor receives asignal stream to create a set of data in the frequency domain, thencreates a first set of input samples in the time domain and at least asecond set of input samples in the time domain. The initiation of thesecond set of input samples time lags the initiation of the first set ofinput samples, thus creating two windows, the commencement of one ofwhich is time-delayed relative to the other. The first and second setsof input samples are then converted to a frequency domain, and frequencydomain data comprising a complex frequency spectrum are outputted foreach of the first and second sets of input samples. In some embodiments,the first and second sets of inputs samples are converted to thefrequency spectrum using at least one of a DFT and a FFT or othertransform. In yet other embodiments, optional transforms to detectfrequency modulation may be applied to the time-delayed windows. In someembodiments a taper or windowing function may be applied to the windowsin the time domain.

In some embodiments, the applied transforms may not output complexdomain data. For example, application of a discrete cosine transform(DCT) tends to result in the output of real data not in the complexdomain.

As is evident, the described pre-processing methods: (i) introduce theconcept of a time lag between windows that allows one to perform CSPEand (ii) may utilize various transforms of the type that are typicallyapplied to perform frequency modulation detection. By “time lag” it ismeant that a second window starts and ends later than the start and endof the first window in an overlapping way. This time lag mimics thehuman brain's ability to store information.

In accordance with one or more embodiments, the single channel superresolution module 208 may be configured to obtain higher frequencyaccuracy to permit and use singlet representation methods to extractcomponents of the original signal such as the signal 202. The singlechannel super resolution module 208 may be configured to use thefollowing inputs such as to facilitate the extraction of components fromthe signal 202. The single channel super resolution module 208 mayrequire input information such as at least two sets of frequency domaindata (FDAT (A) and FDAT (B)) as generated by the single channelpre-processor 204, one or more parameters that may have been used whileapplying a tapering function to the sample window A and the samplewindow B, super-resolved analysis of the transform of the windowingfunction at a resolution that is much finer than the DFT or FFTtransformation and the like. This information can be pre-computedbecause the functional form of the windowing function is known a prioriand can be analyzed to generally any desired degree of precision. Inaddition, the single channel super resolution module 208 may require twoadditional sets of frequency domain data FDAT (B_up) and FDAT(B_down),as generated by the single channel pre-processor 204 for detection ofthe frequency modulation in the signal 202. Optionally, the singlechannel super resolution module 208 may use additional super-resolvedanalysis windows for detection and characterization of amplitudemodulation and joint frequency/amplitude modulation.

FIG. 5 illustrates a method 500 for generating high accuracy frequencyestimates such as to enable the extraction of a set of signalcomponents. The single channel super resolution module 208 may beconfigured to utilize an input 502 that may include the two sets offrequency domain data (FDAT (A) and FDAT (B)) and the analysis window.At step 504, the single channel super resolution module 208 may beconfigured to calculate the complex spectral phase evolution to generatehigh resolution frequencies for subsequent signal extraction. At step508, oscillator peaks in the complex Spectrum (FDAT(A) or FDAT(B)) areidentified such as to generate a list of oscillator peaks 510. Theoscillator peaks may be defined as the projection of an oscillator intothe frequency domain and may be identified as local maxima at some stagein the processing process.

In an example, at step 512, the CSPE behavior of nearby points in thecomplex spectrum (FDAT(A) or FDAT(B)) may be tested for each of theidentified local maxima such as to choose an oscillator peak. Thetesting may facilitate in determining whether the behavior of nearbypoints in the complex spectrum is consistent with the observed behaviornear the peaks in the complex spectrum. Such approach may be applied toretain well-behaved peaks and reject inconsistent peaks. Similarly, foreach individual modulating underlying frequency, the CSPE behavior ofnearby points in the complex Spectrum may be tested such as to determineif they evolve in a manner that is consistent with the observedmodulation behavior near the peaks. In an example, peak rejectioncriteria may be applied to discriminate targeted maxima generated by themain lobe of oscillators from non-targeted maxima generated by otherphenomena such as unwanted noise or side lobes of oscillators. Further,extraction of targeted maxima by a variety of selection criteria may beprioritized. The variety of selection criteria may include but is notlimited to, magnitude selection, frequency selection, psychoacousticperceptual model based selection, or selection based on identificationof frequency components that exhibit a harmonic or approximate harmonicrelationship.

At step 514, one or more singlet representation methods may be used suchas to generate an output. The one or more singlet representation methodsmay include determining the amplitude, phase, and optionally amplitudeand frequency modulation of the oscillator peak 518 at step 520. Inaddition, the one or more singlet representation methods may includegeneration of the updated oscillator peak 522 and update of the spectrumdata at step 524. The method may include removing the contribution ofthe oscillator peak from FDAT (A) and FDAT (B), and this may be done forany type of oscillator peak, including AM modulating and FM modulatingoscillator peaks. The removal of the contribution may extend beyond theregion of the maxima in FDAT(A) or FDAT(B) and separate out the smearedinterference effect of the oscillator on other signal components thatare present. Such type of removal process is a non-local calculationthat may be enabled by the super-resolution analysis of the previousprocessing steps. Further, the singlet representation method may includeconsistent handling of the aliasing of signal components through theNyquist frequency and through the DC (zero-mode) frequency.

At step 528, a determination is made as to whether the process iscompleted. That is to say, the determination of completion of theprocess may include whether an adequate number of targeted maxima areidentified, signal components are prepared for tracking, and/oraggregation into coherent groups, and/or separation and selection,and/or re-synthesis. The single channel super resolution module 208 maybe configured to repeat the processing steps using the spectrum data 530if it is determined that the process is not completed. The method 500proceeds to 532 if it is determined that the process is completed and at532, oscillator peaks 534 are outputted for example, displayed to auser.

FIG. 6 illustrates a method 600 for generating a high accuracy frequencyand AM and FM modulation estimates such as to enable the extraction of aset of signal components. The method 600 may require two additional setsof frequency domain data FDAT (B_up) and FDAT(B_down) when compared tothe data sets as required by the method 500. The additional sets offrequency domain data can enable the detection of AM and/or frequencymodulation within the original signal 202. At step 602, the method 600may perform CPSE on complex spectrum data such as FDAT(A), FDAT(B), FDAT(B_up) and FDAT (B_down). At step 604, an oscillator peak list may becreated and at 608, oscillator peak is chosen using the techniques asdisclosed in 508 and 512 of the method 500 respectively. At step 610,the method 600 may be configured to include one or more singletrepresentation techniques such to extract the components from the signal202. These techniques are further disclosed in the description withreference to advanced singlet fit process. The method 600 may proceed tostep 612 where a determination is made regarding completion of theprocess. On completion, at step 614, the method 600 may output theoscillator peaks.

Thus, in accordance with certain exemplary and non-limiting embodiments,taking the inputs and implementing the methods described herein, aprocessor receives a first set and a second set of frequency domaindata, each having a given, or “fundamental,” transform resolution, andthe processor performs complex spectral phase evolution (CSPE), asfurther described herein, on the frequency domain data to estimatecomponent frequencies at a resolution at very high accuracy, suchaccuracy being typically greater than the fundamental transformresolution. As used herein, “transform resolution” refers to theinherent resolution limit of a transformation method; for example, if aDFT or FFT is calculated on an N-point sample window taken from datathat was sampled at Q samples per second, then the DFT or FFT wouldexhibit N frequency bins, of which half would correspond to positive (orpositive-spinning) frequency bins and half would correspond to negative(or negative-spinning) frequency bins (as defined by a standardconvention known to those familiar with the field); the highest properlysampled signal that can be detected in this method is a frequency of Q/2and this is divided up into N/2 positive frequency bins, resulting in aninherent “transform resolution” of Q/N Hertz per bin. A similarcalculation can be done for any of the other transformation techniquesto determine the corresponding “transform resolution.” In someembodiments there may further be performed peak selection comprisingidentifying one or more oscillator peaks in the frequency domain data,testing the CSPE behavior of at least one point near at least one of theidentified oscillator peaks to determine well-behaved and/orshort-term-stable oscillation peaks and performing an extraction ofidentified oscillator peaks. In yet other embodiments, one may furtherdetermine the amplitude and the phase of each identified oscillator peakand perform singlet transformation/singlet representation to map from ahigh resolution space to a low resolution space. In yet otherembodiments, one may further perform singlet representation to remove acontribution of each identified oscillator peak from the frequencydomain data.

As used above and herein, the “given,” “original” or “fundamental”transform resolution is the resolution of the transform, such as theFFT, used to provide the input data set of frequency domain data—thatis, the inherent resolution of the transform used as the fundamentalbuilding block of the CSPE. Additional details on the CSPEtransformation itself follow.

The CSPE calculates higher accuracy estimates of frequencies than thoseproduced by a conventional transformation, such as the standard DFT orFFT. Conventional FFT and DFT methods assume that the frequency estimateis located in the center of a frequency bin, whereas CSPE in accordancewith one or more embodiments measures the rotation of complex phase of asignal over time to generate a high-resolution estimate of its locationwithin a frequency bin. References to CSPE throughout this disclosureshould be understood to encompass this capability to estimatecharacteristics of a signal, such as rotation of complex phase, at veryhigh resolution within a frequency bin. In accordance with one or moreembodiments, the CSPE method as disclosed herein may provide for asuper-resolution frequency signal analysis. Generally, N samples areobtained from a signal for example, a digitally sampled signal from amusic file in the .wav format, or an output of an analog-to-digitalconverter that may be attached to any sensor device, or a scan line ofan image in black-and-white or RGB format and the like. A Fouriertransform such as the Discrete Fourier Transform (DFT) or Fast FourierTransform (FFT) is performed on the N samples of the signal (e.g.,samples 1, . . . , N). Similarly, N samples are obtained from atime-delayed snapshot of the signal (e.g., samples τ+1, . . . , τ+N fora time delay τ) and a Fourier transform is applied to these time delayedsamples. The phase evolution of the complex Fourier transform betweenthe original samples and the time-delayed samples is then analyzed.Particularly, the conjugate product of the transforms is obtained (withthe multiply being a Schur or Hadamard product where the multiplicationis done term-by-term on the elements of the first transformed vector andthe complex conjugate of the second transformed vector) and then theangle of this conjugate product is obtained. Using this product andangle information, numerous advantageous applications may be realized.For example, the angle may be compared to the transforms to determinefractional multiples of a period such that the correct underlyingfrequency of the signal may be determined. Once the phase evolution isused to determine the correct signal frequency at much higher resolutionthan is possible with the original transform, it becomes possible tocalculate a corrected signal power value. Further, the power in thefrequency bins of the Fourier transforms may be re-assigned to, amongother things, correct the frequency. In this case, the signal power thathas smeared into nearby frequency bins is reassigned to the correctsource signal frequency.

The CSPE algorithm may allow for the detection of oscillatory componentsin the frequency spectrum of the signal 202, and generally provide animproved resolution to the frequencies which may be in the transform. Asstated above, the calculations can be done with the DFTs or the FFTs.Other transforms, however, can be used including continuous transformsand hardware-based transforms.

As shown in the following example, suppose a signal, s(t), is given anda digitally sampled version of the same signal, {right arrow over(s)}=(s₀, s₁, s₂, s₃, . . . ) is defined. If N samples of the signal aretaken, the DFT of the signal can be calculated by first defining the DFTmatrix. For W=e^(i2π/N) the matrix can be written as:

$ = \begin{bmatrix}1 & 1 & 1 & 1 & \ldots & 1 \\1 & W & W^{2} & W^{3} & \ldots & W^{N - 1} \\1 & W^{2} & W^{4} & W^{6} & \ldots & W^{2{({N - 1})}} \\1 & W^{3} & W^{6} & W^{9} & \ldots & W^{3{({N - 1})}} \\\vdots & \vdots & \vdots & \vdots & \ddots & \vdots \\1 & W^{N - 1} & W^{2{({N - 1})}} & W^{3{({N - 1})}} & \ldots & W^{{({N - 1})}{({N - 1})}}\end{bmatrix}$

Each column of the matrix is a complex sinusoid that is oscillating aninteger number of periods over the N point sample window. In accordancewith one or more embodiments, the sign in the exponential can bechanged, and in the definition of the CSPE, the complex conjugate can beplaced on either the first or second term.

For a given block of N samples, define

${{\overset{\rightarrow}{S}}_{0} = \begin{bmatrix}s_{0} \\s_{1} \\s_{2} \\s_{3} \\\vdots \\s_{N - 1}\end{bmatrix}},{{\overset{\rightarrow}{S}}_{1} = \begin{bmatrix}s_{1} \\s_{2} \\s_{3} \\s_{4} \\\vdots \\s_{N}\end{bmatrix}},$

and in general,

${{\overset{\rightarrow}{s}}_{i} = \begin{bmatrix}s_{i} \\s_{i + 1} \\s_{i + 2} \\s_{i + 3} \\\vdots \\s_{i + N - 1}\end{bmatrix}},$

the DFT of the signal can be computed as

${F\left( {\overset{\rightarrow}{s}}_{i} \right)} = {\begin{bmatrix}1 & 1 & 1 & 1 & \ldots & 1 \\1 & W & W^{2} & W^{3} & \ldots & W^{N - 1} \\1 & W^{2} & W^{4} & W^{6} & \ldots & W^{2{({N - 1})}} \\1 & W^{3} & W^{6} & W^{9} & \ldots & W^{3{({N - 1})}} \\\vdots & \vdots & \vdots & \vdots & \ddots & \vdots \\1 & W^{N - 1} & W^{2{({N - 1})}} & W^{3{({N - 1})}} & \ldots & W^{{({N - 1})}{({N - 1})}}\end{bmatrix}\begin{bmatrix}s_{i} \\s_{i + 1} \\s_{i + 2} \\s_{i + 3} \\\vdots \\s_{i + N - 1}\end{bmatrix}}$

As described above, the CSPE may analyze the phase evolution of thecomponents of the signal between an initial sample of N points and atime-delayed sample of N points. Allowing the time delay be designatedby Δ and the product of F({right arrow over (s)}₁) and the complexconjugate of F({right arrow over (s)}_(i+Δ)), the CSPE may be defined asthe angle of the product (taken on a bin by bin basis, equivalent to the“.*” operator in Matlab, also known as the Schur product or Hadamardproduct) CSPE=≮(F({right arrow over (s)}_(i))⊙F*({right arrow over(s)}_(i))), where the ⊙operator indicates that the product is taken onan element-by-element basis as in the Schur or Hadamard product, and the≮ operator indicates that the angle of the complex entry resulting fromthe product is taken.

To illustrate this exemplary process on sinusoidal data, take a signalof the form of a complex sinusoid that has period p=q+δ where q is aninteger and δ is a fractional deviation of magnitude less than 1, i.e.,|δ|≦1. The samples of the complex sinusoid can be written as follows:

${\overset{\rightarrow}{s}}_{0} = \begin{bmatrix}^{0} \\^{\; 2{\pi \cdot \frac{q + \partial}{N} \cdot}} \\^{\; 2{\pi \cdot 2}{\frac{q + \partial}{N} \cdot}} \\^{\; 2{\pi \cdot 3}{\frac{q + \partial}{N} \cdot}} \\\vdots \\^{\; 2{\pi \cdot {({N - 1})}}{\frac{q + \partial}{N} \cdot}}\end{bmatrix}$

If one were to take a shift of one sample, then Δ=1 in the CSPE, and:

${\overset{\rightarrow}{s}}_{1} = \begin{bmatrix}^{\; 2{\pi \cdot \frac{q + \partial}{N} \cdot}} \\^{\; 2{\pi \cdot 2}{\frac{q + \partial}{N} \cdot}} \\^{\; 2{\pi \cdot 3}{\frac{q + \partial}{N} \cdot}} \\^{\; 2{\pi \cdot 4}{\frac{q + \partial}{N} \cdot}} \\\vdots \\^{\; 2{\pi \cdot N}{\frac{q + \partial}{N} \cdot}}\end{bmatrix}$

which can be rewritten to obtain:

${\overset{\rightarrow}{s}}_{1} = {\begin{bmatrix}^{\; 2{\pi \cdot \frac{q + \partial}{N} \cdot}} \\^{\; 2{\pi \cdot 2}{\frac{q + \partial}{N} \cdot}} \\^{\; 2{\pi \cdot 3}{\frac{q + \partial}{N} \cdot}} \\^{\; 2{\pi \cdot 4}{\frac{q + \partial}{N} \cdot}} \\\vdots \\^{\; 2{\pi \cdot N}{\frac{q + \partial}{N} \cdot}}\end{bmatrix} = {{^{{2\pi} \cdot \frac{q + \partial}{N}}\begin{bmatrix}^{0} \\^{\; 2{\pi \cdot \frac{q + \partial}{N} \cdot}} \\^{\; 2{\pi \cdot 2}{\frac{q + \partial}{N} \cdot}} \\^{\; 2{\pi \cdot 3}{\frac{q + \partial}{N} \cdot}} \\\vdots \\^{\; 2{\pi \cdot {({N - 1})}}{\frac{q + \partial}{N} \cdot}}\end{bmatrix}} = {^{{2\pi} \cdot \frac{q + \partial}{N}}{\overset{\rightarrow}{s}}_{0}}}}$

One determines the conjugate product (again, taken on anelement-by-element basis) of the transforms, the result is:

${{F\left( {\overset{\rightarrow}{s}}_{i} \right)} \odot {F^{*}\left( {\overset{\rightarrow}{s}}_{i + 1} \right)}} = {{^{{- {2\pi}} \cdot \frac{q + \partial}{N}}{{F\left( {\overset{\rightarrow}{s}}_{i} \right)} \odot {F^{*}\left( {\overset{\rightarrow}{s}}_{i} \right)}}} = {^{{- {2\pi}} \cdot \frac{q + \partial}{N}}{{F\left( {\overset{\rightarrow}{s}}_{i} \right)}}^{2}}}$

The CSPE is found by taking the angle of this product to find that:

${\frac{2\pi}{N}{CSPE}} = {{\nless \left( {{F\left( {\overset{\rightarrow}{s}}_{i} \right)} \odot {F^{*}\left( {\overset{\rightarrow}{s}}_{i} \right)}} \right)} = {2{\pi \cdot \frac{q + \delta}{N}}}}$

If this is compared to the information in the standard DFT calculation,the frequency bins are in integer multiples of

$\frac{2\pi}{N},$

and so the CSPE calculation provided information that determines thatinstead of the signal appearing at integer multiples of

$\frac{2\pi}{N},$

the signal is actually at a fractional multiple given by q+δ. Thisresult is independent of the frequency bin under consideration, so theCSPE may allow an accurate determination of underlying frequency nomatter what bin in the frequency domain is considered. In looking at theDFT of the same signal, the signal would have maximum power in frequencybin q−1, q, or q+1, and if δ≠0, the signal power would leak to frequencybins well outside the range of bins. The CSPE, on the other hand, mayallow the power in the frequency bins of the DFT to be re-assigned tothe correct underlying frequencies that produced the signal power. Inaccordance with one or more embodiments, the definition of the W matrix,the columns on the right are often interpreted as “negative frequency”complex sinusoids, since

$\begin{bmatrix}1 \\W^{N - 1} \\W^{2{({N - 1})}} \\W^{3{({N - 1})}} \\\vdots \\W^{{({N - 1})}{({N - 1})}}\end{bmatrix} = \begin{bmatrix}1 \\W^{- 1} \\W^{- 2} \\W^{- 3} \\\vdots \\W^{1}\end{bmatrix}$

similarly the second-to-last column is equivalent to

$\quad\begin{bmatrix}1 \\W^{- 2} \\W^{- 4} \\W^{- 6} \\\vdots \\W^{2}\end{bmatrix}$

The phrase ‘negative frequency components’ as used herein thedescription may indicate the projection of a signal onto the columnsthat can be reinterpreted in this manner (and consistent with thestandard convention used in the field).

In accordance with one or more embodiments, the oscillator peakselection process as used in the methods 400 and 500 of the description,may facilitate in identification of maxima in the frequency domainspectra that are main-lobe effects of oscillators, and determination ofan optimal order in which to extract the oscillator peaks from thefrequency domain data. In an example, the oscillator peak selectionprocess may include converting the complex frequency data stored in FDAT(A) to an amplitude. The amplitude of an element of FDAT (A) is theabsolute value of the complex value of that element. The amplitude of anelement of the FDAT (A) may also be referred herein to as spectrumamplitude (A).

The oscillator peak selection process can include identifying localmaxima in the spectrum amplitude (A). In an example, an element atlocation n is a local maximum if the amplitude at the location n isgreater than the amplitude of the element at location n−1 and theamplitude of the element at location n+1. Further, the local maxima maybe tested such as to identify main-lobe effects of the oscillators thatare referred herein to as the oscillator peaks. For example, theamplitude of the local maxima may be tested against a minimum thresholdvalue. In another example, proximity of the CSPE frequency correspondingto the location of the local maxima is determined with respect to thecenter of the FFT frequency bin corresponding to that location. If theCSPE frequency is not proximate enough, this may signify that the localmaximum is a side-lobe effect of an oscillator or is a noise-inducedpeak. However, if the amplitude of the local maxima is greater than acertain threshold, the local maxima may be considered to be asignificant peak regardless of earlier tests and may be constructed froma group of oscillators.

The oscillator peak selection process can include determining an orderin which to extract oscillator peaks from the FDAT (A) and FDAT (B).Higher priority peaks are chosen using selection criteria appropriatefor a given application; that is, for example, certain types of higherorder peaks are typically more characteristic of desired signals, ratherthan noise, in given situation. Peaks may be chosen by, among othertechniques, magnitude selection, a psycho-acoustic perceptual model(such as in the case of signal extraction for speech recognition orspeech filtering), track duration, track onset times, harmonicassociations, approximate harmonic associations or any other criteriaappropriate for a given application.

In accordance with one or more embodiments, the CSPE high resolutionanalysis may be configured to convert tone-like signal components tostructured (e.g., line) spectra with well-defined frequencies, while thenoise-like signal bands do not generally take on structure. As such, thesignal may be substantially segregated into the tone-like and thenoise-like components. To select oscillator peaks, in embodiments aseries of steps may be employed. For example, firstly, the CSPE analysismay test the complex spectral phase evolution behavior of nearby pointsin the complex spectrum for each individual underlying frequencydetected such as to determine if they evolve in a manner that isconsistent with the observed behavior near the peaks in the complexspectrum. Further criteria may be applied to retain well-behaved peaksand reject poorly behaved (e.g., inconsistent) peaks.

In an example, the CSPE analysis may be configured to conduct adeconvolution analysis for the each consistent, well-behaved peak suchas to determine the amplitude and phase of the underlying signalcomponent that produced the measured FFT or DFT complex Spectrum. Thedata obtained from the high resolution frequency analysis can be used toprioritize the components of the signal in order of importance; forexample, priority in the case of recognition of speech signals in anoisy environment may be based on perceptual importance or impact onintelligibility. A psychoacoustic perceptual model (PPM) may be providedin the Unified Domain such that independent computations for eachchannel of data may not have to be computed separately, and the UnifiedDomain PPM may give information that may be used to give priority tospecific components in the multi-channel data. In an example, theUnified Domain PPM may be used to give emphasis to signals coming from aspecified direction or range of directions. Accordingly a UnifiedPsychoacoustic Perceptual Model (UPPM) is provided that incorporates theeffects of spectral, spatial and temporal aspects of a signal into onealgorithm. This algorithm may be embodied in hardware or performed insoftware.

In accordance with one or more embodiments, the UPPM computation may beseparated into three steps. The first step may include a high resolutionsignal analysis that may distinguish between tone-like and noise-likesignal components. The second step may include calculation of thecoherency groups of signal components based on frequency, sound pressurelevel, and spatial location, with each coherency group providing a “unitof intelligibility” that may be enhanced. Further, the interference andseparability of the coherency groups may be calculated and projected tocreate a Coherency Surface in the Unified Domain. In an example, theCoherency Surfaces may be utilized to create a surface that is definedover the entire spatial field. In addition, Coherency Curves can beobtained with a transformation from the Unified Domain for stereo audiosignals, left and right channel. Thus, a traditional single-channelprocessing techniques can still be performed on a signal. At any time, amulti-channel signal can be transformed back into the Unified Domain ora signal in the Unified Domain can be transformed into a multi-channelsignal (or a single-channel signal) for signal processing purposes.

In accordance with one or more embodiments, the singlet representationmethod may include a set of operations that can identify the parametersof an oscillator from frequency domain data, or can generate frequencydomain data using the parameters of an oscillator. Various steps in thesinglet transformation process in accordance with one or moreembodiments may include calculating the normalized shape of theprojection of an oscillator in the frequency domain. Further, the stepsmay include calculating the magnitude and phase of an oscillator byfitting the calculated spectrum to a set of frequency data andcalculating the magnitude and phase of a low frequency oscillator,accounting for interference effects caused by aliasing through DC. Inaddition, the steps may include adding or subtracting an oscillator'sfrequency domain representation to or from frequency domain data,accounting for aliasing though Nyquist and DC. In accordance with one ormore embodiments, complex analysis methods may be employed to furthercharacterize an oscillator peak's frequency and amplitude modulationwithin a single FFT window. These complex algorithms are discussedfurther in detail in the description.

In accordance with one or more embodiments, a normalized shape of theoscillator's projection in the frequency domain may be calculated usingan input including a high resolution frequency domain version of theanalysis window used in the single channel pre-processor 204 and ahigh-accuracy frequency estimate of an oscillator peak, as created byCSPE. The high resolution frequency domain version of the analysiswindow used in the single channel pre-processor 204 may also be referredherein to as FWIN. The FWIN is the frequency domain representation of ahigh-resolution version of the analysis window used in the singlechannel pre-processor 204 such as to apply a taper to the sample windowA and sample window B. The FWIN may be longer than the original analysisby a factor of 16 or 32. This factor is called the ‘upsample’ rate. Inother embodiments, the high-resolution version of the analysis windowmay be known exactly through a mathematical functional representation.

If it is determined that the frequency of the oscillator is in thecenter of the FFT bin, the shape of the oscillator's projection matchesa down-sampled version of FWIN, and can be created by first choosing apointer from FWIN at its center, then choosing points at intervals ofthe upsample rate. If the frequency is not in the center of the FFT bin,the shape of the oscillator may correspond to a subset of FWIN slightlyoffset in frequency from those points. In accordance with one or moreembodiments, the shape of an oscillator's projection into the frequencydomain may be created using a method that may include calculating thedistance in frequency between the center of the FFT bin corresponding tothis location and the CSPE frequency. The method may further includechoosing a first sample from FWIN at the FWIN's center plus the offsetas calculated above. The method may include a calculation step choosingsamples from FWIN at predetermined intervals based on upsample rate. Forexample, if the calculated offset corresponds to five bins in FWIN, theupsample rate is 16, and FWIN's center bin corresponds to 32769, thenthe FWIN bins chosen may be: [ . . . , 32741, 32757, 32773, 32789, 32805. . . ]. In an example, the number of bins that may be chosen may dependon a user-defined parameter of the system. The output of this step is aset of complex samples chosen from FWIN and these complex samples may bereferred herein to as the oscillator peak normalized spectrum. Thesesamples may have inaccurate amplitude and phase.

In accordance with one or more embodiments, accurate amplitude and phaseof the complex samples chosen from FWIN may be calculated ondetermination of the peak shape of the oscillator. In an example, theamplitude and phase calculation may be accomplished by fitting theoscillator's shape to a set of frequency domain data, typically storedin FDAT (A) and this phase may also be referred herein to as a Fitamplitude phase that may need an oscillator peak normalized spectrum, anoscillator peak's high accuracy frequency as calculated by CSPE, and aset of frequency domain data, typically FDAT (A). Further, the methodmay include solving for the magnitude and phase rotation that fits thespectrum data and multiplying sampled normalized spectrum by newmagnitude and phase.

On estimation of the frequency of a signal component, an accurateapproximation of the contribution of that signal component to themeasured spectrum of a signal can be determined. In one or moreembodiments, this follows from a property of the discrete FourierTransform when applied to signals that are not centered in the middle ofa frequency bin. This process follows from the properties of convolutionand windowing. In other embodiments, where the high-resolution versionof the analysis window may be known exactly as mathematical functionalrepresentation, the convolutional properties may be calculated in acontinuous fashion.

In accordance with one or more embodiments, when a signal is analyzed, afinite number of samples may be selected, and a transform may becomputed. As an example and not as a limitation, a DFT may be applied onthe signal. However, other transforms having similar properties and arewell known to researchers familiar with the art may be applied on thesignal. The transform of the window of data is generally preceded by awindowing step, where a windowing function, W(t), is multiplied by thedata, S(t). Suppose W(t) is called the analysis window (and later thewindows of data can be reassembled using the same or different synthesiswindows). Since the data is multiplied by the window in the time domain,the convolution theorem states that the frequency domain representationof the product of W(t)*S(t) exhibits the convolution of the transforms,Ŵ(f) and Ŝ(f), where the notation indicates that these are thetransforms of W(t) and S(t), respectively. If the high resolutionspectral analysis reveals that there is a signal component of magnitudeM₀ at a frequency f₀, then the convolution theorem implies an existenceof a contribution centered at f₀ that is shaped like the analysiswindow, giving a term essentially of the form M₀Ŵ(f−f₀). In a discretespectrum, such as the spectrum calculated by the discrete Fouriertransform, there is a finite grid of points that result in a sampledversion of the spectrum. Thus, the contribution centered at f₀ describedabove is sampled on the finite grid points that are integer multiples ofthe lowest nonzero frequency in the spectrum. Equivalently, if thediscrete Fourier transform is calculated for N points of data that hasbeen properly sampled with a sample rate of R samples/sec, then thehighest frequency that is captured is the Nyquist frequency of R/2 Hzand there will be N/2 independent frequency bins. This provides a lowestsampled frequency of (R/2 Hz)/(N/2 bins)=R/N Hz/bin. In addition, allother frequencies in the discrete Fourier transform are integermultiples of R/N.

Because of the relationship between the analysis window transform, Ŵ(f)and the spectral values that have been sampled onto the frequency gridof the discrete transform, such as the discrete Fourier transform, it ispossible to use knowledge of Ŵ(f), along with the measured sample valueson the grid points nearest to f₀, to calculate a good estimate of themagnitude, M₀. To calculate this value, the nearest frequency grid pointto f₀ (f_(grid)) is identified, a difference Δf=f₀−f_(grid) iscalculated, and the magnitude value of the transform of the signal atthat grid point M_(grid) is calculated. The true magnitude can then becalculated from the following relation

$\frac{M_{grid}}{{\hat{W}\left( {{- \Delta}\; f} \right)}} = \frac{M_{0}}{{\hat{W}}_{\max}}$

In an example, ∥Ŵ_(max)∥ is taken to mean the maximum magnitude of thetransform of the analysis window, and is generally normalized to 1.Also, the transform of the analysis window is generally symmetric, sothe sign of Δf generally does not matter. These relations can be adaptedfor more unusual windowing functions by those skilled in the art bymanipulation of the basic convolution relation. Assuming a fixedresolution to the knowledge of Ŵ(f), Ŵ(f) can be sampled on afine-scaled grid that is 2 times, 4 times, 8 times, 16 times, 32 times,or 64 times, or N times finer than the resolution of the frequency gridin the DFT. In this case, the difference value Δf is calculated to thenearest fraction of a frequency bin that may correspond to thefine-scaled grid. For example, if the fine scaled grid is 16 times finerthan the original frequency grid of the transform, then Δf is calculatedto 1/16 of the original frequency grid. The desired fine-grainedresolution is dependent on the particular application and can be chosenby one skilled in the art.

In accordance with one or more embodiments, the phase of the true signalmay be adjusted on estimation of the true signal frequency and magnitudeso that the signal may align with the phases that are exhibited by thediscrete frequency spectrum. So, if φ_(grid) represents the phase angleassociated with the magnitude M_(grid), and φ_(win) represents the phaseangle of Ŵ(−Δf), then the analysis window must be rotated by an amountthat is equal to φ_(rot)=φ_(grid)−φ_(win). Once this is done, all of theinformation about the signal component is captured by the values of f₀,M₀, and φ_(rot). As a result, reconstruction of the signal componentneeds a representation of the analysis window, Ŵ(f), shifting of therepresentation to frequency f₀, rotating it by angle φ_(rot), andmultiplying it by magnitude M₀ (assuming the analysis window has maximummagnitude equal to 1, otherwise multiply by a factor that scales thewindow to magnitude M₀). The output of fit amplitude and phase includesthe oscillator peak's true amplitude and phase, and the oscillatorpeak's scaled spectrum.

In accordance with one or more embodiments, an accurate amplitude andphase may be calculated for a low frequency oscillator on determinationof the peak shape of the oscillator. In such cases of the low frequencyoscillator, the effect of aliasing through DC may interfere with thecomplex conjugate projection of that oscillator. Because of thisinterference, it is difficult to measure true amplitude and phase of theoscillator using conventional techniques. The methods presented here inaccordance with one or more embodiments represent an innovation thatsolves the problem of calculating the true amplitude and phase of thelow frequency oscillator. In an example, the method may include using aninput that may include a low frequency oscillator peak's normalizedspectrum, a low frequency oscillator peak's high accuracy frequency ascalculated by CSPE, and a set of frequency domain data, typically FDAT(A) such as to determine the true amplitude and phase of the lowfrequency oscillator. In certain embodiments, the method may include aniterative step wherein CSPE frequency values are varied through a rangeof values to achieve an improved match to the oscillator's spectrum.

In signal processing applications, if data is sampled too slowly, thenan aliasing problem at high frequencies may be present. Interferencealso exists at extremely low frequencies and will be referred to hereinas the interference through DC problem. This problem occurs when finitesample windows are used to analyze signals. The windowing function usedin the sampling is intimately involved, but the problem can occur in thepresence of any realizable finite-time window function. To state theproblem clearly, assume that a signal of frequency f₀ is present and isclose to the DC or 0 Hz frequency state. If such a signal is sampledover a finite-time window W(t), then, as discussed above, the frequencyspectrum of the signal is equal to the convolution in the frequencydomain of a delta function at frequency f₀, with the Fourier transformof the windowing function, which is designated as Ŵ(f). In a discreteformulation, the result is then projected onto the grid of frequenciesin the discrete transform, e.g., onto the frequency grid of the FastFourier Transform (FFT). Since the transform of the windowing functionis not infinitely narrow, the spectrum has power spilling over intofrequency bins other than the one that contains f₀. In fact, thetransform of the windowing function extends through all frequencies, sosome of the signal power is distributed throughout the spectrum causinga pollution of nearby frequency bins from the spillover of power.Depending on the windowing function, the rate at which Ŵ(f) falls tozero varies, but for most windows used in practice, e.g., Hanningwindows, Hamming windows, Boxcar windows, Parzen windows and many othersknown to those skilled in the art, there is significant spillover beyondthe bin that contains f₀. This spillover or smearing effect is importantthroughout the spectrum of a signal, and when two signal components areclose in frequency, the interference from the spillover can besignificant. However, the problem becomes acute near the DC bin, becauseany low frequency signal has a complex conjugate pair as its mirrorimage on the other side of DC. These complex conjugate signals are oftenconsidered as “negative frequency” components, but for a low frequencysignal, the pairing guarantees a strong interference effect. However,the complex conjugate nature of the pairing allows for a solution of theinterference problem to reveal the true underlying signal and correctfor the interference, if a good estimate of the frequency can beachieved. The methods described herein address the problem of theconventional methods. The method may include considering the spectrum atf₀, and the measured spectral value at f₀ as a reflection of acontribution from the “positive frequency” component, which will bedesignated as Ae^(iσ) ¹ , and a contribution from the mirror image or“negative frequency” component, Be^(iσ) ² . Since the Be^(iσ) ²contribution comes from the negative frequencies at −f₀, thecontribution at +f₀ is taken from the conjugate of the analysis windowŴ*(f). If Ŵ*(f) is assumed to be defined so that it is centered at f=0,then the contribution from the negative frequency component comes at adistance 2f₀ from the center of Ŵ*(f). Consequently, if a highresolution estimate of the frequency f₀, is obtained, then thecontributions to the measured spectral value at +f₀ from positive andnegative frequencies can be determined. The method can include settingthe phase to be 0 at both the +f₀ and −f₀ positions. When set in thisposition, the values for Ae^(iσ) ¹ and Be^(iσ) ² are known completely,and so the difference σ₁−σ₂ is obtained. In addition, when the phase is0, the signal components in the +f₀ and −f₀ positions are real, so thecomplex conjugate spectrum from the negative frequency is in the samerelative phase position as the spectrum in the positive frequencies.However, when the phase becomes different from 0, the relative phasevalues must rotate in the opposite sense, so that if the phase at +f₀ isset to φ, then the phase at −f₀ must be set to −φ to maintain thecomplex conjugate pairing. This means that in the zero phaseorientation, the contributions Ae^(iσ) ¹ and Be^(iσ) ² have a relativephase difference of σ₁−σ₂, but as the phase orientation at +f₀ is set toφ, the phase orientation at −f₀ counter-rotates and becomes set to −φ,so the contribution Be^(iσ) ² must counter-rotate by the same amount.Thus, in any phase orientation, the net contribution at a givenfrequency is a combination of rotated and counter-rotated versions ofAe^(iσ) ¹ and Be^(iσ) ² , and these sums trace out an ellipse. Also,since the major axis of the ellipse will occur when Ae^(iσ) ¹ andBe^(iσ) ² are rotated into alignment, this occurs when the rotationangle is

$\theta = {\frac{1}{2}\left( {\sigma_{1} - \sigma_{2}} \right)}$

and the sum of the rotated and counter-rotated versions becomes

${{^{\frac{- }{2}{({\sigma_{1} - \sigma_{2}})}}\left( {A\; ^{{\sigma}_{1}}} \right)} + {^{\frac{}{2}{({\sigma_{1} - \sigma_{2}})}}\left( {B\; ^{{\sigma}_{2}}} \right)}} = {\left( {A + B} \right){^{\frac{}{2}{({\sigma_{1} + \sigma_{2}})}}.}}$

As a result, the major angle occurs when the rotation andcounter-rotation put the terms into alignment at an angle that is theaverage of the phase angles. The position of the minor axis can besimilarly determined, since it occurs after a further rotation of π/2radians. Thus, the sum of the rotated and counter-rotated versions forthe minor axis becomes

${{^{\frac{}{2}}{^{\frac{- }{2}{({\sigma_{1} - \sigma_{2}})}}\left( {A\; ^{{\sigma}_{1}}} \right)}} + {^{{- }\frac{}{2}}{^{\frac{}{2}{({\sigma_{1} - \sigma_{2}})}}\left( {B\; ^{{\sigma}_{2}}} \right)}}} = {\left( {A - B} \right){^{\frac{}{2}{({\sigma_{1} + \sigma_{2} + \pi})}}.}}$

The method may further include facilitating parameterization of theellipse so that the angular orientation can be determined in astraightforward manner. To start with, consider an ellipse with majoraxis on the x-axis and of magnitude M, and let S be the magnitude of theminor axis. The ellipse can then be parameterized by τ→(M cos τ,S sinτ), and by specifying a value for r, any point on the ellipse can bechosen. If τ gives a point on the ellipse, and the angular position, ρ,of the point in polar coordinates (since this will correspond to thephase angle for the interference through DC problem), can be found fromthe relation

${\tan \; \rho} = {\frac{S\; \sin \; \tau}{M\; \cos \; \tau} = {\frac{S}{M}\tan \; {\tau.}}}$

When this form of parameterization is applied to the interferencethrough DC problem, the ellipse formed by rotated and counter-rotatedsums of Ae^(iσ) ¹ and Be^(iσ) ² is rotated so that the major and minoraxes align with the x- and y-axes, and then the measured spectrum isexamined to determine the actual angle exhibited by the resultantspectral components. The resultant angle from the measured spectrum islabeled Ω. Since the major axis is at

${\Delta = {\frac{1}{2}\left( {\sigma_{1} + \sigma_{2}} \right)}},$

a further rotation is needed to put the resultant at angle Ω. Therefore,τ corresponding to Ω−Δ needs to be determined, and in an example, isobtained using the following relation:

${\tan \left( {\Omega - \Delta} \right)} = {\frac{A - B}{A + B}\tan \; \tau}$

-   -   provided as the result:

$\tau = {\tan^{- 1}\left( {\frac{A + B}{A - B}{\tan \left( {\Omega - \Delta} \right)}} \right)}$

The method may further include recognizing that the relations above aredetermined solely from knowledge of the frequencies and complexconjugate relationship at the +f₀ and −f₀ positions in the spectrum. Allof the analysis was determined from the relative magnitudes of thetransform of the windowing function. The relative magnitudes will remainin the same proportion even when the signals are multiplied by anamplitude value. Therefore, the recreation of the true measured spectrummay require selecting the true amplitude value from the spectrum andthen rescale the sum of the rotated and counter-rotated contributions sothat they equal the amplitudes exhibited by the measured spectralvalues. The final result is a highly accurate measure of the trueamplitude of the signal at +f₀, so that when the spectrum isreconstructed with the windowing function Ŵ(f) positioned at +f₀, andits mirror-image, complex conjugate pair, Ŵ*(f), placed at −f₀, theresulting sum that includes the interference through the DC bin will bea highly accurate reconstruction of the true, measured signal spectrum.

The above analysis has focused on the interaction at the +f₀ and −f₀positions in the spectrum and a similar analysis can be conducted at anyof the affected frequencies to derive an equivalent result. The analysisat the +f₀ and −f₀ positions is for illustrative purpose since thesignal is concentrated there, and in practice generally gives thehighest signal to noise ratio and most accurate results. The output offit amplitude and phase for low frequency oscillators is a low frequencyoscillator peak's true amplitude and phase, and a low frequencyoscillator peak's scaled spectrum.

In one or more examples, it may be determined that the estimate of +f₀may not be sufficiently accurate. In these cases, it is possible to varythe value of +f₀ over a range of frequencies and continue to iterate theprocess until a desired accuracy is reached and is discussed further indetail in the description in a section [00150] below.

In accordance with one or more embodiments, some or all oscillator peaksthat are fit using the low-frequency method are tested and corrected forerror. The method of testing and correcting the low frequency oscillatorpeak error may include subtracting an oscillator peak from spectrum towhich it was fit and calculating the residual spectrum. If the residualspectrum near the center of that oscillator peak is above a threshold,the method may include modifying the CSPE frequency at intervals oneither side of the original spectrum, and repeating low frequencyamplitude and phase calculations. Accordingly, the method may includeusing the oscillator peak with the lowest residual error. The methoddescribed in this section may be used in the process of frequency andamplitude modulation detection.

In accordance with one or more embodiments, interference by anoscillator aliasing across DC or Nyquist should be accounted for whenremoving or adding an oscillator peak to or from frequency data such asto prevent the incorrect identification of peaks or re-synthesis ofpeaks. In an example, this can be accomplished by implementing a methodfor adding or subtracting the complex conjugate of the portion thatwraps through DC or Nyquist in addition to the primary addition orsubtraction. In an example, the method may include using an input thatmay include an oscillator peak's high resolution frequency as calculatedby CSPE, an oscillator peaks' scaled spectrum, and a set of frequencydomain data such as to perform oscillator peak addition and subtraction.The method can include identifying the location of the oscillator peakin the frequency domain data and dividing the oscillator peak into aprimary region and a tail region if the oscillator peak is situated suchthat it is bisected by either the DC frequency or the Nyquist frequency.The tail region is the portion that lies in the negative frequenciesbetween DC and −Nyquist (negative Nyquist) (where we adopt theconvention that half of the frequencies in the complex FFT aredesignated positive (or positive-spinning) frequencies and half of thefrequencies are designated negative (or negative-spinning) frequencies.

In an example, the method can include adding the primary region to theinput frequency domain data and adding the complex conjugate of the tailregion to the input frequency domain data when an additive operation isperformed to prevent the incorrect identification of peaks orre-synthesis of peaks. Otherwise, the method can include subtracting theprimary region from the input frequency domain data and subtracting thecomplex conjugate of the tail region from the input frequency domaindata to prevent incorrect identification of peaks or re-synthesis ofpeaks. The method may output a modified set of the frequency domain datareceived as input to this step.

In accordance with one or more embodiments, the output of thesingle-channel super-resolution methods may include a set of parametersdescribing individual oscillator components. Each set may include theinformation used to accurately reconstruct that oscillator in the singlechannel re-synthesis methods. In a preferred embodiment, the informationmay include frequency, amplitude, and phase related informationcorresponding to the oscillator component.

In an example, the multi-channel pre-processor 210 may be configured, inaccordance with one or more embodiments, to prepare multi-channel timedomain signal data that may be processed by the multi-Channel CSPE superresolution techniques. In an example, as an input, a multi-channeltime-domain signal may be fed to the multi-channel pre-processor 210.The input may be a live feed or a recorded file. In another example,single-channel data streams may be processed by the single-channelpre-processor.

The multi-channel pre-processor 210 may be configured to follow the samemethods described as discussed previously for the single-channelpreprocessor in 204, but the methods may be repeated for multiplechannels of data. In an example, the multi-channel pre-processor 210 mayperform a method for each channel of input signal in accordance with oneor more embodiments. The method may include filling a sample window withn sequential samples of input signal for that channel. In an example,the sequential sample windows may be configured to overlap with eachother such that the size of the sample window and number of samples thatthe sample window overlaps with subsequent and previous sample windowsmay be specified by the user in a parameter file. The size and number ofoverlapping sample window may also be set as part of a software orhardware implementation. For exemplary purposes a sample window may bedefined, hereinafter referred to as Sample Window (A).

The method may further include creating a second, time-delayed samplewindow. For exemplary purpose, the second sample window may hereinafterbe referred as Sample Window (B). The sample window A and the samplewindow B may be offset in time such that the sample window B lags thesample window A. Sample Window (B) lags Sample Window (A). The methodmay further include creating two more time-delayed sample windows iffrequency modulation detection is desired. The additional sample windowsmay contain the same samples as the sample window B, but the additionalwindows may be processed differently. The additional windows mayhereinafter be referred to as (B_up) and (B_down) for exemplary purpose.The detection of frequency modulation may include applying a ‘ModulationPullback Operator’ to the (B_up) and (B_down) sample windows. This maybe accomplished via a Hadamard product. For example, for the (B_up)sample window, a Modulation Pullback Operator for positive frequencymodulation may be used. Further, for the (B_down) sample window, aModulation Pullback Operator for negative frequency modulation may beused. The method may further include applying an analysis window, ortaper, to both, the sample window A and the sample window B separately.This may be accomplished via the Hadamard product, as discussedpreviously. In an example, the frequency modulation detection mayinclude applying the analysis window to the (B_up) and (B_down) samplewindows. The method may further include converting both the taperedsample window A and the tapered sample window B to the frequency domainusing a DFT or FFT. For exemplary purposes, the frequency domain outputmay hereinafter be referred to as FDAT_channel_X (A) and FDAT_channel_X(B), where X is the identifier of the channel. Further, if frequencymodulation detection may be desired, the FDAT_channel_X (B_up) andFDAT_channel_X (B_down) windows may be created using the same process asdiscussed previously for the (B_up) and (B_down) sample windows.

In an example, an output of the multi-channel pre-processor 210 mayinclude two sets of data per frame, such that each data set may havebeen converted to the frequency domain via the Fast Fourier Transform(FFT) technique or any other related frequency transform technique. Foreach channel, the second set may lag the first set by a small number ofsamples, corresponding to a slight time delay. For the exemplary purposeof description, these data sets may be referred as FDAT_channel_(—)0(A), FDAT_channel_(—)0 (B) . . . FDAT_channel_N (A), and FDAT_channel_N(B). In an example, if frequency modulation detection is desired, twoadditional frequency domain data sets may be created for each channel.These may be hereinafter exemplarily referred to as the FDAT_channel_X(B_up) and FDAT_channel_X (B_down). FDAT_channel_X (B_up) andFDAT_channel_X (B_down) may be the frequency domain representations ofthe time delayed samples that may be contained in the sample window Band that may have had a Modulation Pullback Operator applied to thembefore conversion to the frequency domain. The FDAT_channel_X (B_up) mayhave a positive frequency Modulation Pullback Operator applied, and theFDAT_channel_X (B_down) may have a negative frequency ModulationPullback Operator applied.

In accordance with an exemplary and non-limiting embodiment, apreprocessor may receive a plurality of signal streams to create a setof data in the frequency domain. The frequency domain data may comprisea plurality of sample windows, or “data sets”. For the purpose ofdescription, the “Sample window” may refer to a window of n samples thatmay be taken from an original time series data. Each of the plurality offrequency domain data sets may then be used to create a first data setand a second data set wherein the initiation of the second data set timemay lag the initiation of the first data set, and each of the pluralityof sample data sets may be converted to a frequency domain and outputtedas a complex frequency spectrum for each of the first and second datasets. In some examples, each corresponding first data set/window andsecond data set/window may be converted to the frequency spectrum, suchas by using a conventional transform, such as a FFT, DCT, or any othersuch transform.

In an example, a multi-channel super resolution module may be defined.The multi-channel super resolution module may be configured to obtain ahigher frequency accuracy to permit the use of singlet transforms toextract components of an original signal. In an example, the input ofthe multi-channel super-resolution module may include two sets forfrequency domain data for each channel from the multi-channelpre-processor 210. The data set may be hereinafter referred to as anFDAT_channel_(—)0 (A) and an FDAT_channel_(—)0 (B) . . . FDAT_channel_N(A), and FDAT_channel_N (B), where the channel is specified aschannel_(—)0 up to channel_N, and the frequency data is specified as (A)for non-time-delayed data and (B) for time-delayed data.

The input may further include parameters describing the analysis windowused when applying a taper to the sample window A and the sample windowB. In an example, if frequency modulation is desired, the input mayfurther include two additional sets of frequency domain data, a dataFDAT (B_up) and a data FDAT (B_down), as generated by the single channelpre-processor. In an example, the input may further include optionaladditional super-resolved analysis windows for detection andcharacterization of frequency and amplitude modulation.

FIG. 7 illustrates a method 700 for unified domain super resolution. Themethod illustrates by way of example, performing signal decomposition inthe Unified Domain by decomposing into discrete objects such as steadytones, noise-like elements, transient events, and modulatingfrequencies. The method 700 in accordance with one or more embodimentsmay be an extension of the single-channel super-resolution methods.

The method may include, at 702, performing unified domain transform andunified domain complex spectral phase evolution (CSPE) on complexspectral phase evolution frequencies obtained from the plurality ofinput channels. The input channels may include the channels input to themulti-channel super resolution module, such as the channelsFDAT_channel_(—)0 (A), FDAT_channel_(—)0 (B) . . . FDAT_channel_N (A),and FDAT_channel_N (B) as discussed previously. The method 700 mayfurther include using the singlet transform methods to remove thecontribution of the oscillator peak from FDAT_channel_(—)0 (A),FDAT_channel_(—)0 (B) . . . FDAT_channel_N (A), FDAT_channel_N(B). Thismay be done at 704, by creating a list of oscillator peaks from thetransformed channel data. Further, from the list of oscillator peaks, at708, an oscillator peak may be chosen using a peak selection process.The peak selection process may include identifying oscillator peaks. Thepeak selection process may further include applying peak rejectioncriteria to discriminate targeted maxima generated by the main lobe ofoscillators from non-targeted maxima generated by other phenomena suchas unwanted noise or side lobes of oscillators. The targeted maxima maythen be prioritized based on a plurality of factors including amagnitude and frequency of separation, an application of apsychoacoustic model, or tracker state information that may be used toprioritize peak selection. The method 700 may further include, at 710,using singlet transform methods to identify the amplitude and phase ofthe oscillator peak in each channel. The oscillator peak's informationmay be saved for an output from the process. The method 700 may furtherinclude, at 710, making a determination if the process may be complete.If the process is complete, at 712, the oscillator peak informationsaved previously may be provided as an output of performing the method700. Alternatively, if at 712 it is determined that the process ofidentifying oscillator peaks is not complete, the method 700 may berepeated.

In an example, the method 700 may further include preparing signalcomponents for tracking and/or filtering and/or re-synthesis. In analternate embodiment, the method for unified domain super resolution maybe used in conjunction with amplitude and frequency modulationdetection. FIG. 8 illustrates such a method 800, which incorporatesadditional amplitude and frequency modulation detection at 802, apartfrom the steps included in the method 700. At 802, if amplitude and/orfrequency modulation detection is desired, a technique involving theAdvanced Fit Process is used.

s may be used to identify the frequency and/or amplitude modulation ofthe oscillator peaks that may have been chosen as previously discussedfor method 700.

The unified domain representation of the spectrum data in accordancewith one or more embodiments may be calculated using the techniques of

which will be discussed in the following description.

Unified Domain may be a representation of multi-channel signals as asingle channel of data. There may be lossless transformation thatconverts a multi-channel signal into a Unified Domain. As a result, asignal in the Unified Domain may be processed as a whole, rather thanseparately processing the individual channels. In an example, even whena signal is transformed into the Unified Domain, all of the signal'sinformation about the magnitudes, frequencies, and spatial componentrelated to a signals location may be retained. The transformation of thesignal may be an invertible technique such that a signal in the UnifiedDomain may be reverted back to a multi-channel signal, such as asurround-sound signal, or a stereo signal of an RGB signal. In anexample, the Unified Domain transformation may include a feature suchthat the original, multi-channel signal may be converted to arepresentation where a single magnitude component is multiplied by amatrix from the special unitary group, SU(N), where N represents thenumber of channels in the original data signal.

In an example, the process of converting to the Unified Domain (UD) maybegin when a multi-channel signal stream may be converted to a singlechannel signal stream in the Unified Domain. A transformation may beutilized to perform the conversion. The transformation may includeretaining information about the magnitudes, frequencies, internalphases, and spatial locations of the signal components of each channelwhile placing the information in a single “signal”. Further, thetransformation may include using a stream of matrices rather than asingle, 1-dimensional stream of data samples. The UD transformation maybe an invertible technique as the UD representation involves a singlemagnitude component multiplied by an element of the complex SpecialUnitary group for N-channels (SU(N)). In some examples, the UD matrixmay be taken from the Unitary Group U(n). The SU(N) group may berepresented in many ways. For the purposes of transforming amulti-channel signal, the structures of complex matrices may beemployed. In an example, stereo input may be represented in UD. Sincestereo input includes two channels, such that N=2, accordingly, therepresentation in the Unified Domain may be provided as a singlemagnitude component multiplied by a 2×2 complex matrix. Moreparticularly, the transformation of a multi-channel audio stream may berepresented as:

T:C ^(N)

mag*SU(N)≡U ^(N)

[audio_(ch0) audio_(ch1) . . . audio_(chN−1) ]

U ^(N)

where the magnitude may be a function of frequency, N may represent thenumber of input channels, and U represents the Unified Domain.

For a conventional two channel audio stream (such as Left/Right) therepresentation may become:

[L R]

U ²

This representation may include a one-to-one mapping between the twochannel audio stream and the representation as a stream of matrices inthe UD and the transformation may be lossless. Any manipulations done inone domain may have an equivalent counterpart in the other domain.Persons skilled in the art may appreciate that a number of processingtechniques may be performed on a signal in the Unified Domain that mayprove to be advantageous. For example, a process applied to a signal maybe performed faster since the process may only have to be performed oncein the Unified Domain, while the process would otherwise have to beperformed separately for each sub-channel. Furthermore, Unified Domainmanipulations have the advantage of operating on all of the channels atthe same time, thus keeping the channels synchronized without the needfor additional synchronization processes to be performed.

In accordance with exemplary and non-limiting examples, a processor maybe configured to receive a plurality of channels, each comprising afirst set and a second set of frequency domain data having a transformresolution. The plurality of channels may be combined into a unifieddomain representation and complex spectral phase evolution (CSPE) may beperformed on the unified domain representation to estimate componentfrequencies at a resolution or accuracy greater than the fundamentaltransform resolution. In such examples, the mathematics discussed abovemay apply uniformly as CSPE turns the plurality of channels into arepresentation in the Unified domain. For example, instead of a rightand left channel, CSPE may render a single matrix form representationincluding all of the inputted channels.

In other examples, further performing peak selection may be performedcomprising identifying one or more oscillator peaks in the unifieddomain representation and testing the CSPE behavior of at least onepoint near at least one identified oscillator peak to retainwell-behaved peaks. These identified peaks may then be extracted inprioritized fashion. In other examples, singlet representation may beperformed to identify amplitude and phase of each identified oscillatorpeak. In yet other examples, singlet representation may be performed toremove a contribution of each identified oscillator peak from theunified domain representation.

In an example, unified domain CPSE methods may be defined. In anexample, a method may include performing a processing step on a signalin the Unified Domain that may include performing a high resolutionfrequency analysis. The high resolution frequency analysis may be anextension of the 1-dimensional CSPE transformation discussed previously.As in the 1-dimensional case, the phase evolution of the components of asignal in the Unified Domain may be analyzed between an initial sampleof N points and a time delayed sample of N points. From this comparison,a fractional multiple may be obtained that is representative of thespatial location where the signal components actually appear. As aresult, the correct underlying frequency and estimated spatial locationfor the signal may be determined. To correct the underlying frequencypresent in the sampled signal, the information may be utilized tore-assign signal power in the frequency bins of the transform utilizedto obtain the high resolution frequency analysis.

In accordance with one or more examples, one process that may beutilized to manipulate a signal in the Unified Domain may be a highresolution frequency analysis and the process may be implemented as amatrix-based version of the Complex Spectral Phase Evolution (CSPE)method. As a result, the transformation may in certain examples, forexample, give signal accuracies on the order of 0.01 Hz for stablesignals at CD sample rates analyzed in approximately 46 ms windows. Incertain other examples, signal accuracies of 0.01 Hz, 0.001 Hz or evenfiner accuracies may result. The CSPE high resolution analysis may becapable of converting tone-like signal components to line spectra withwell-defined frequencies, while the noise-like signal bands do not takeon structure. As such, the signal may be substantially segregated intotone-like and noise-like components. Further processing may be utilizedto, such as, detect if there is the presence of a transient signalcomponent or an amplitude- or frequency-modulating signal component in aframe of sample data or test for, and aggregate, harmonic groupings offrequencies. Persons skilled in the art may appreciate that theprocessing may be performed on an entire signal (e.g., an entire audiosignal) or portions of a signal. As such, a windowing step may beprovided at any point in the process. For example, frames of data may betaken directly from the multi-channel data stream or from the data inthe Unified Domain.

In an example, the UD transformation may provide a way to analyze datasimultaneously in multiple channels, such as might be present in musicfor stereo music with two channels or surround sound music with multiplechannels. In a similar example, one may consider image and video data tobe composed of multiple channels of data, such as in the RGB format withRed, Blue and Green channels. Thus, the multi-channel signal may berepresented in the form of a one-dimensional magnitude vector in thefrequency domain, multiplied by a vector of matrices taken from theSpecial Unitary Group, SU (n). Accordingly, a more particulartransformation of a multiple channel signal to a signal in the UnifiedDomain may occur as follows.

In one illustrative example, the input data may be stereo musiccontaining 2 channels of data designated Left and Right, and the resultmay be a magnitude vector multiplied by a vector of matrices from theSpecial Unitary Group of dimension 2, SU(2). A transformation process toachieve the above mentioned conversion of stereo music to the resultantmagnitude vectors may include a plurality of steps. The first step mayinclude selecting a window of music data and transform it to thefrequency domain using a transformation such as the Discrete FourierTransform (DFT). As a result of performing the step, a representation ofthe signal in discrete frequency bins may be obtained. In an example, Nsamples may be selected in the window of data. Consequently N frequencybins may be obtained. Alternatively, there may be variations of thetransforms known to those skilled in the art that may alter the numberof frequency bins.

The frequency domain transformation may result in 2 channels of(generally) complex frequency information. Thus, each frequency bin maybe viewed as a complex vector with 2 elements. These elements may thenbe multiplied by a complex matrix taken from the group SU (2), resultingin a single magnitude component. This magnitude component may be storedwith the matrix as the representation of the stereo music.

In an example, the transformation process may be representedmathematically as follows:

left channel: {right arrow over (S)} _(L) =s _(0L) ,s _(1L) ,s _(2L), .. .

right channel: {right arrow over (S)} _(R) =s _(0R) ,s _(1R) ,s _(2R), .. .

To convert to the frequency domain, the following mathematicaloperations may be performed:

{right arrow over (F)} _(L)=DFT({right arrow over (s)} _(L))

{right arrow over (F)} _(R)=DFT({right arrow over (s)} _(R))

The group elements may be represented in a plurality of ways. Forexample, for the SU(2) matrices for 2 channels of data therepresentation may take the form as represented below:

$U = \begin{bmatrix}{^{- {\varphi}_{1}}\cos \; \sigma} & {^{- {\varphi}_{2}}\sin \; \sigma} \\{{- ^{{\varphi}_{2}}}\sin \; \sigma} & {^{{\varphi}_{1}}\cos \; \sigma}\end{bmatrix}$

In an example, the angles with components of the frequency domainvectors may be identified as follows. Let the j^(th) complex componentof {right arrow over (F)}_(L) be designated as a_(j)+ib_(j)=r_(Lj)e^(iφ)¹ and the j^(th) complex component of {right arrow over (F)}_(R) bedesignated as c_(j)+id_(j)=r_(Rj)-e^(iφ) ² . The complex frequencycomponents may then be identified with the elements of the (KS note:this must appear as SU(2) with no gaps or separations or carriagereturns inserted) SU(2) matrix for the j^(th) frequency bin by setting

${\cos \; \sigma} = {r_{Lj}/\sqrt{\sqrt{r_{Lj}^{2} + r_{Rj}^{2}}}}$

and

${{\sin \; \sigma} = {r_{Rj}/\sqrt{\sqrt{r_{Lj}^{2} + r_{Rj}^{2}}}}},$

and the phase variables may be the same φ₁ and φ₂ values. If the SU(2)matrix is multiplied by a 2-vector of the frequency components for thej^(th) frequency bin, then the result may be a single magnitude vector:

${\left\lbrack U_{j} \right\rbrack \begin{bmatrix}F_{Lj} \\F_{Rj}\end{bmatrix}} = \begin{bmatrix}\sqrt{\sqrt{r_{Lj}^{2} + r_{Rj}^{2}}} \\0\end{bmatrix}$

The SU (2) matrices may be preferably unitary and may have inversematrices, such that, all of the information may be contained in themagnitude vector and the U matrix. Thus, a new representation for thetwo channel data may be provided that may contain all of the informationthat was present in the original:

${\sqrt{\sqrt{r_{Lj}^{2} + r_{Rj}^{2}}}\left\lbrack U_{j} \right\rbrack} = {\sqrt{\sqrt{r_{Lj}^{2} + r_{Rj}^{2}}}\begin{bmatrix}{^{- {\varphi}_{1}}\cos \; \sigma_{j}} & {^{- {\varphi}_{2}}\sin \; \sigma_{j}} \\{{- ^{{\varphi}_{2}}}\sin \; \sigma_{j}} & {^{{\varphi}_{1}}\cos \; \sigma_{j}}\end{bmatrix}}$

In one or more examples, once the data is represented in the UnifiedDomain representation, the previously represented two independentchannels of music, that is to say, the right and the left frequencies,may be represented in the Unified Domain as a single magnitude vectormultiplied by a complex matrix from SU(2). The transformation may beinverted easily, so it may be possible to change back and forth in aconvenient manner.

In the one or more examples discussed above, a majority of the signalprocessing operations that may be used in processing multi-channelsignals may be computed in the Unified Domain. So, in one application,the front end processing may use a calculation of the Complex SpectralPhase Evolution (CSPE). The Unified CSPE may be calculated by convertinga window of data to the Unified Domain. The representation for thatwindow may be called Λ₁. Further, a time-shifted window of data to theUnified Domain may be represented as Λ₂. The Unified CSPE may thenrequire a calculation of Λ₁⊙Λ₂*, where the operator ⊙ is configured totake the component-wise product (also known as the Schur product orHadamard product) of the matrices over all of the frequency bins, andthe * indicates that the complex conjugate is taken. In order to obtainthe remapped frequencies of the CSPE in the Unified Domain, thearguments of the complex entries in the Unified CSPE may be calculated.

In an example, the traditional signal processing functions may beadvantageously reformulated so that they may be computed in the UnifiedDomain. In an example, there may be a mathematical equivalence betweenthe Unified Domain and the usual representations of data in thefrequency domain or the time domain. When coupled with the remapping ofthe frequencies in the Unified CSPE, it may become possible to considerthe signal components as having a spatial position and internal phaserelationships. This may be done, such that, in the case where the inputdata is stereo audio with right and left channels, by associating thespatial effect of the stereo audio to operate over a field spanning anangle of approach to the listener. In this view, a signal component thatmay occur with a given value of a may be viewed as occurring at angle ain the stereo field, with a magnitude given by the magnitude componentderived from the Unified Domain representation magnitude values.Furthermore, the internal phase angles of the 2 channels may bepreserved in the φ₁ and φ₂ values assigned to that signal component.

In an example, the music on the left and right channels may be composedof two components, with frequencies f₀ and f₁. When the components maybe converted to the Unified Domain and processed with the Unified CSPE,these signals may be associated with their magnitudes, spatialpositions, and internal phases so f₀

|f₀|, σ₀, φ₀₁ and φ₀₂ and for the second signal, the association is f₁

|f₁|, σ₁, φ₁₁ and φ₁₂. Then, determination of the coherency surface maybe adapted to have a spatial component. For example, if a signalcomponent such as f₀, would have a 1-dimensional masking effect overnearby frequencies that is given by the masking function G (f₀: f), thenthis masking effect may be extended to the unified domain, the coherencysurface function would pick up a spatial component related to theangular separation between the signal components, and one can representone embodiment of this as a coherency function H(f₀; f,σ)=G(f₀;f)*cos(σ−σ₀), where the cosine function represents the spatialcomponent. Similarly, a coherency function may be derived for everysignal component and a global coherency surface defined over the entirespatial field of the data may be found, for example, by taking the sumof the coherency functions at a given point in the spatial field, or themaximum of the coherency functions at a given point in the spatial fieldor the average of the coherency functions at a point in the spatialfield or any of a number of other selection rules for the coherencyfunctions at a point in the spatial field. Further, other spatialfunctions than the cosine function may be utilized as well as functionsthat drop off faster in the spatial direction or functions that fall offslower in the spatial direction.

In an example, the process of converting to the Unified Domain,calculation of high-resolution Unified CSPE information, and calculationof Coherency surfaces in the Unified Domain, may provide the possibilityto jointly consider all of the components that make up a multi-channelsignal and process them in a consistent manner. In alternative examples,other refinements and examples of the applicability of the signalprocessing algorithms may be made. For example, the CSPEsuper-resolution algorithm may be applied more generally than just to asingle signal component. Accordingly, the CSPE algorithm may be used toresolve many signals components provided there is some separationbetween the signal frequencies. When multiple signals may be present,the super-resolution of the frequencies may be most accurate nearspectral frequency bins that may be dominated by an individual signalcomponent, and the regions of the spectrum that are away from the signalcenters may be generally remapped to the nearest dominant signalfrequency. For example, for a signal composed of three sinusoids thesignals do not lie in the center of frequency bins. In this example, thealgorithm may be configured to successfully recalculate the trueunderlying frequencies with good accuracy. FIG. 9 illustrates agraphical representation of this process (see 910). The original FFTspectrum is shown as line 911 and the remapped spectrum is shown as line912; the remapped spectrum is effectively a line spectrum. For thisexample, the exact frequencies (in frequency bin numbers) are28.7965317, 51.3764239, and 65.56498312, while the estimated frequenciesare 28.7960955, 51.3771794, and 65.5644420. If these spectra werecalculated from music sampled at CD sampling rates of 44100 samples/sec,the fundamental transform resolution of each frequency bin would beapproximately 21.53 Hz/bin, so the measured signals are accurate toapproximately ±0.001 bins, which is equivalent to ±0.02153 Hz. However,the real-world music data may not be as clean and stable. Thus, theaccuracy of the computed high-resolution spectrum may be affected suchas by the presence of nearby interfering signals, modulations of thefrequencies, and noise-like signals that have a broadband spectrum. Insuch examples, the high-resolution analysis may give signal accuracy ofthe order of 0.1 Hz for any signal component that may be relativelystable over the sample window. An example is given for a window of datataken from a track by Norah Jones and the remapped spectrum appears insignal 920, where the original signal is line 922 and the remappedsignal is line 921. In an example of an alternate variation of thealgorithm, a similar resolution may be provided for a linearlymodulating signal component while returning a high-resolution estimateof the initial signal frequency in the window, along with the modulationrate. This may be affected by changing the CSPE to include amultiplication by a complex vector that counteracts the modulation by ameasured amount (the pull-back operator). This may be discussed furtherin the sections on frequency modulation discussed in the supportingdescription.

The CSPE technique may also be utilized for real signals in addition tocomplex signals, as real functions may be expressed as the sum of acomplex function and its complex conjugate function. For example, for areal sinusoid with period p=q+δ where p is an integer and δ is afractional deviation of magnitude less than 1, i.e. |δ|≦1, withamplitude “a” and arbitrary phase, the samples of a real sinusoid may bewritten as linear combinations of complex sinusoids, such as thefollowing (here j=√{square root over (−1)}):

${\overset{\rightarrow}{s}}_{0{(n)}} = {{\frac{a}{2}^{j\frac{2{\pi {({q + \delta})}}}{N}n}} + {\frac{a}{2}^{{- j}\frac{2{\pi {({q + \delta})}}}{N}n}}}$

and the one sample shift would be:

${\overset{\rightarrow}{s}}_{1{(n)}} = {{\frac{a}{2}^{j\frac{2{\pi {({q + \delta})}}}{N}n}^{j\frac{2{\pi {({q + \delta})}}}{N}}} + {\frac{a}{2}^{{- j}\frac{2{\pi {({q + \delta})}}}{N}n}^{{- j}\frac{2{\pi {({q + \delta})}}}{N}}}}$

if

$D = ^{j\; \frac{2\pi {({q + \delta})}}{N}}$

is defined, the vectors may be written as:

${\overset{->}{s}}_{0{(n)}} = {{\frac{a}{2}D^{n}} + {\frac{a}{2}D^{- n}}}$${\overset{->}{s}}_{1{(n)}} = {{\frac{a}{2}D^{n}D} + {\frac{a}{2}D^{- n}D^{- 1}}}$

In this example, the DFT of each one of these vectors may then be:

${F\left( {\overset{->}{s}}_{0} \right)} = {F\left( {{\frac{a}{2}D^{n}} + {\frac{a}{2}D^{- n}}} \right)}$${F\left( {\overset{->}{s}}_{0} \right)} = {{\frac{a}{2}{F\left( D^{n} \right)}} + {\frac{a}{2}{F\left( D^{- n} \right)}}}$${F\left( {\overset{->}{s}}_{1} \right)} = {F\left( {{\frac{a}{2}D^{n}D} + {\frac{a}{2}D^{- n}D^{- 1}}} \right)}$${F\left( {\overset{->}{s}}_{1} \right)} = {{\frac{a}{2}{{DF}\left( D^{n} \right)}} + {\frac{a}{2}D^{- 1}{F\left( D^{- n} \right)}}}$

The CSPE may be computed using the complex product F({right arrow over(s)}₀)⊙F({right arrow over (s)}₁) of the shifted and unshiftedtransforms, where the product operator ⊙ may be defined as the complexproduct taken element-by-element in the vector:

$\begin{matrix}{{{F\left( {\overset{->}{s}}_{0} \right)}{F^{*}\left( {\overset{->}{s}}_{1} \right)}} = {\left\lbrack {{\frac{a}{2}{F\left( D^{n} \right)}} + {\frac{a}{2}{F\left( D^{- n} \right)}}} \right\rbrack \odot \left\lbrack {{\frac{a}{2}{{DF}\left( D^{n} \right)}} + {\frac{a}{2}D^{- 1}{F\left( D^{- n} \right)}}} \right\rbrack^{*}}} \\{= {{\left( \frac{a}{2} \right)^{2}\left\lbrack {{F\left( D^{n} \right)} + {F\left( D^{- n} \right)}} \right\rbrack} \odot \left\lbrack {{D^{*}{F^{*}\left( D^{n} \right)}} + {{DF}^{*}\left( D^{- n} \right)}} \right\rbrack}}\end{matrix}$

The product may be expanded to obtain the following

${{F\left( {\overset{->}{s}}_{0} \right)}{F^{*}\left( {\overset{->}{s}}_{1} \right)}} = {\left( \frac{a}{2} \right)^{2}\begin{bmatrix}{{D^{*}{{F\left( D^{n} \right)} \odot {F^{*}\left( D^{n} \right)}}} +} \\{{{{DF}\left( D^{n} \right)} \odot {F^{*}\left( D^{- n} \right)}} +} \\{{D^{*}{{F\left( D^{- n} \right)} \odot {F^{*}\left( D^{n} \right)}}} +} \\{{{DF}\left( D^{- n} \right)} \odot {F^{*}\left( D^{- n} \right)}}\end{bmatrix}}$

The above equation may be simplified to produce:

${{F\left( {\overset{->}{s}}_{0} \right)}{F^{*}\left( {\overset{->}{s}}_{1} \right)}} = {\left( \frac{a}{2} \right)^{2}\begin{bmatrix}{{D^{*}{{F\left( D^{n} \right)}}^{2}} +} \\{{{{DF}\left( D^{n} \right)} \odot {F^{*}\left( D^{- n} \right)}} +} \\{{D^{*}{{F\left( D^{- n} \right)} \odot {F^{*}\left( D^{n} \right)}}} +} \\{D{{F\left( D^{- n} \right)}}^{2}}\end{bmatrix}}$

In an example, the above simplified equation may be viewed as a sum ofthe CSPE for a “forward-spinning” or “positive-frequency” complexsinusoid and a “backward-spinning” or “negative-frequency” complexsinusoid, plus interaction terms.

The first and the last terms in the sum may be the same as previouslydiscussed CSPE calculations, but instead of a single complex sinusoid,there may be a linear combination of two complex sinusoids. Further, thecontributions to the CSPE from these two terms may representhighly-concentrated peaks positioned at q+δ and −(q+δ), respectively.The interaction terms may have some properties that may decrease theaccuracy of the algorithm if not handled properly. As will be shownbelow, the bias introduced by the interaction terms may be minimized bywindowing the data. Additionally, the interaction terms, r, may besimplified as follows:

Γ=[DF(D ^(n))⊙F*(D ^(−n))+D*F(D ^(−n))⊙F*(D ^(n))]

Γ=2*Re[DF(D ^(n))⊙F*(D ^(−n))]

F(D^(n)) may be, for example, a peak concentrated at frequency positionq+δ, and that F(D^(−n)) may be a peak concentrated at frequency position−(q+δ), and that the product may be taken on an element-by-elementbasis, (so Γ≈0 for a number of cases).

The data provided in the exemplary scenario discussed above may beanalyzed using an analysis window, including but not limited to aHanning window, a Hamming window, or a rectangular window, or any otherstandard windowing function. Further, the measured spectrum may be foundby convolving the true (that is to say, delta-like) sinusoidal spectrumwith the analysis window. For example, if a rectangular window (such as,the boxcar window) is used, the leakage into nearby spectral bins may besignificant and may be of sufficient strength to produce significantinteraction terms. The interaction terms may cause the magnitude squaredterms (that is to say, the terms in ∥∥² brackets) to interfere. Toreduce the chance of significant interaction terms, another analysiswindow known in the art may be utilized so that the leakage may beconfined to the neighborhood of q+δ, and −(q+δ), so the Γ≈0 case is themost common situation. Further, after the CSPE is calculated, thefrequencies may be reassigned by extracting the angle information. Forthe positive frequencies (such that where k>0), it may be determinedthat:

$\begin{matrix}{f_{CSPEk} = \frac{{- N}\; {\measuredangle \left( {{F_{k}\left( {\overset{->}{s}}_{0} \right)}{F_{k}^{*}\left( {\overset{->}{s}}_{1} \right)}} \right)}}{2\pi}} \\{= \frac{{- N}\; {\measuredangle \left( {\left( \frac{a}{2} \right)^{2}{{F_{k}\left( D^{n} \right)}}^{2}^{{- j}\; \frac{2\pi {({q + \delta})}}{N}}} \right)}}{2\pi}} \\{= \frac{- {N\left( {- \frac{2\pi \left( {q + \delta} \right)}{N}} \right)}}{2\pi}}\end{matrix}$ f_(CSPEk) = (q + δ)

For the negative frequencies (k<0), the opposite value, f_(CSPEK)=−(q+δ)may be determined. Consequently, in the case of real signals (such asmay be the case when Γ≈0), all of the power in the positive frequenciesmay be remapped to q+δ, and all of the power in the negative frequenciesmay be remapped to −(q+δ). Such a result may be substantiallyindependent of the frequency bin and may allow for extremely accurateestimates of frequencies.

In an example, CSPE may be performed for real sinusoids that have beenwindowed with an analysis window. CPSE may then be generalized, forexample, to include the effects of windowing by defining the basictransform to be a windowed transform. For exemplary purpose, data may bewindowed before computing the DFT. Further, for the purpose of exemplarydiscussion, an arbitrary analysis window, A (t), and its sampled versionA, may be defined. The transforms may be performed as has been discussedpreviously. Further, the analysis window may be pre-multiplied by thefunction illustrated as below:

F({right arrow over (s)} ₀)

F({right arrow over (A)}⊙{right arrow over (s)} ₀)≡F _(W)({right arrowover (s)} ₀)

where the W subscript indicates that a windowed transform may beutilized.

Thus, in the presence of windowing, the following may be obtained:

${{F_{w}\left( {\overset{->}{s}}_{0} \right)}{F_{w}^{*}\left( {\overset{->}{s}}_{1} \right)}} = {\left( \frac{a}{2} \right)^{2}\begin{bmatrix}{{D^{*}{{F_{w}\left( D^{n} \right)}}^{2}} +} \\{{2\mspace{11mu} {Re}\left\{ {{{DF}_{W}\left( D^{n} \right)} \odot {F_{W}^{*}\left( D^{- n} \right)}} \right\}} +} \\{D{{F_{W}\left( D^{- n} \right)}}^{2}}\end{bmatrix}}$

The transform may enable minimizing the leakage into nearby frequencybins and further, reducing the interference terms to be negligible inmost cases.

In accordance with some exemplary and non-limiting embodiments, in aunified domain model/super-resolution model for signal processing, aninteraction among non-orthogonal AM/FM elements may be determined in afrequency-changing signal. Such determination may be made “trackeraware” so that an interaction tracker may be configured to look at thehistory of tracklets as they are evolving to make a consistentdetermination between the AM and FM components.

In an example, a method for performing modulation detection through anadvanced fit process may be defined. For the purpose of discussion ofthe method, an assumption regarding the conventional Fourier basedanalysis methodology may be made. The assumption may describe that theconventional Fourier based analysis operates in a manner that anyoscillator peak may be produced by a stable sinusoid during the time ofa single analysis window, with a constant frequency and amplitude. Formany applications, however, it may be necessary to detect changes infrequency and/or amplitude within a single analysis window. Suchdetection may be made by considering in combination or in isolation, oneor more of the techniques as may be discussed below.

In an example, an amplitude modulation (AM)/frequency modulation (FM)detection technique using high resolution window (HRW) creation may bedefined. The technique may include a singlet transformation processincluding applying a high resolution, frequency domain version of theanalysis window to the time-domain samples to characterize theoscillator peak that may be analyzed. For the purpose of description,the high-resolution frequency domain version of the analysis window maybe referred to as an (HRW).

In an example, the singlet transformation process may be used tocharacterize the oscillator peaks that may not be constant in amplitudeand/or frequency within the sample window. In order to do so, an HRWwith the corresponding amplitude and/or frequency modulation may be usedfor analysis. Such an HRW designed for amplitude modulation mayhereinafter be referred to as an (AM HRW) for the purpose ofdescription. For example, to analyze an oscillator peak that may be theresult of a sinusoid that increased in amplitude during the samplewindow, it may be compared to an HRW where the analysis window used tocreate the HRW may be multiplied by the same increasing amplitude priorto conversion to the frequency domain. In a similar example, to analyzean oscillator peak that is modulating in frequency, an HRW where theanalysis window is multiplied by a Modulation Creation Operator for thecorresponding frequency modulation rate prior to conversion to thefrequency domain may be used. Such an HRW may be hereinafter referred toas an (FM HRW) for the purpose of description.

The detection techniques discussed above may be combined to analyze theeffects of a sinusoid with both amplitude and frequency modulation. Suchan HRW may be hereinafter referred to as an (AM/FM HRW) for the purposeof description.

FIG. 10 illustrates an example of a method for creating thehigh-resolution AM/FM windows. The method 1000 includes starting theprocess of high-resolution window creation with a copy of an originalanalysis window, such as is illustrated by the Window function 1002. Themethod 1000 may further include, at 1008, multiplying the analysiswindow 1002 by the desired amplitude modulation 1004, using such as aHadamard product, for creating a window for the analysis of amplitudemodulation 1012. The method 1000 may further include, at 1014,multiplying the window 1012 by the frequency Modulation CreationOperator 1010 with the appropriate modulation amount to create thewindow for the analysis of frequency modulation 1018. The FrequencyModulation Creation Operator (FMCO) may be configured to transform asinusoid that is stable in frequency to one that is modulating infrequency. The method 1000 may further include, at 1020, padding thewindow 1018 to the desired length. In a preferred example, the desiredlength may be 16 or 32 times the original length of the sample window.Further, at 1020, an FFT or DFT may also be performed to the transformthe analysis to the frequency domain. The transformation may result in ahigh resolution window (A) 1022, as illustrated in the example of FIG.10.

The method 1000, may also be performed alternatively by repeating thesteps 1008 till 1020 by using time shifted AM window 1024 and/or timeshifted FM window 1030, that may be obtained by shifting the AM window1004 and the FM window 1010 by the appropriate shifting factors for thetime delay used when preparing the Sample time delayed high resolutionWindow (B) 1034 in the pre-processor.

In an example of a method for amplitude modulation (AM) detection,amplitude modulation may be detected by using the Singlet TransformationMethod to compare various AM HRWs, each of which may have a different AMenvelope applied in the time domain, as discussed previously. An AM HRWmay be closest in shape in the frequency domain to an oscillator peakcreated from a sinusoid that has a similar amplitude modulation. Thus,the amplitude modulation of the original signal may be detected byselecting the AM HRW with the lowest residual error

FIG. 11 illustrates an example of a method 1100 for frequency modulationdetection. The method 1100 includes, at 1134, using the originaltime-domain audio samples 1102 in the pre-processor to create twoadditional (B) windows, the frequency domain sample window (B_UP) 1110,and the frequency domain sample window (B_DOWN) 1112. The additionalwindows may be created by, applying the Frequency Modulation PullbackOperator (FMPO) for a positive modulation to one (B) window, at 1134 c,and call the window the (B_up) window 1110. This may be accomplished viaa Hadamard product. Similarly, the method 1030 allows for the creationof Frequency Modulation Creation Operators. Similarly, the method 1100may include, at 1134 d, applying the Frequency Modulation PullbackOperator (FMPO) for a negative modulation to the other (B) window, andcall it the (B_down) window 1112. This may also by accomplished via theHadamard product. The method 1100 may further include, at 1138 (a-c),performing three Complex Spectral Phase Evolutions (CSPEs), as discussedin the description for Single Channel Super-Resolution Module, anon-modulation CSPE, at 1138 a, of the (A) window and the (B) window; anup modulating CSPE, at 1138 b, of the (A) window and the (B_up) window;and a down modulating CSPE, at 1138 c, of the (A) window and the(B_down) window. In accordance with certain non-limiting examples,taking the inputs and implementing the methods described herein, aprocessor may be configured to receive a first set and a second set offrequency domain data, each having a given, or “fundamental,” transformresolution, and the processor may further be configured to performcomplex spectral phase evolution (CSPE), as further described herein, onthe frequency domain data to estimate component frequencies at aresolution at very high accuracy, such that the accuracy may betypically greater than the fundamental transform resolution. As usedherein, “transform resolution” may refer to the inherent resolutionlimit of a transformation method; for example, if a DFT or FFT iscalculated on an N-point sample window taken from data that was sampledat Q samples per second, then the DFT or FFT may exhibit N frequencybins, of which half would correspond to positive (or positive-spinning)frequency bins and half would correspond to negative (ornegative-spinning) frequency bins (as may be defined by a standardconvention known in the art). The highest properly sampled signal thatmay be detected in this method may include a frequency of Q/2 that maybe divided up into N/2 positive frequency bins, resulting in an inherent“transform resolution” of Q/N Hertz per bin. A similar calculation maybe done for any of the other transformation techniques to determine thecorresponding “transform resolution.” In some examples there may furtherbe performed peak selection comprising identifying one or moreoscillator peaks in the frequency domain data, testing the CSPE behaviorof at least one point near at least one of the identified oscillatorpeaks to determine well-behaved and/or short-term-stable oscillationpeaks and performing an extraction of identified oscillator peaks. Inother examples, further the amplitude and the phase of each identifiedoscillator peaks may be determined and a singlet transformation/singletrepresentation may be performed to map from a high resolution space to alow resolution space. In other examples, a singlet representation may beperformed to remove a contribution of each identified oscillator peakfrom the frequency domain data.

As used above and herein, the “given,” “original” or “fundamental”transform resolution is the resolution of the transform, such as theFFT, that may be used to provide the input data set of frequency domaindata—that is, the inherent resolution of the transform used as thefundamental building block of the CSPE. Additional details on the CSPEtransformation may be described in the following description.

In an example, performing the CPSE at 1138 a-1138 c may result in thegeneration of three CSPE windows, a CSPE window 1114, a CSPE_UpModulation window 1118, and a CSPE_down modulation window 1120. Once anoscillator peak may be selected, the ‘flatness’ of the area around thepeak in the CSPE, CSPE_up and CSPE_down may be analyzed. A signal withpositive frequency modulation may have a flatter area around the peak inthe CSPE_up, a signal with negative frequency modulation may have aflatter area around the peak in the CSPE_down, and a signal with arelatively low amount of frequency modulation may have a flatter areaaround the peak in the CSPE. For the purpose of description, the‘flatness’ may refer to a plot of estimated frequency (or its equivalentmeasure, the effective rotation in complex space of the transforms forthe A and B windows) such that the frequency bins near the oscillatorpeak map to a nearly constant value. The method 1100 may furtherinclude, at 1140 (a-c), subtracting the values in the left and right ofpeak from the CSPE window 1114, the CSPE_up window 1118 and theCSPE_down window 1120, to identify the width in CSPE 1122, the width inCSPE_up 1124 and the width in CSPE_down 1128 respectively, of thefrequency modulation. If frequency modulation is detected, the valuescalculated at 1140, that is to say the values 1120-1124, may be used at1142 to interpolate the exact amount of frequency modulation. As aresult, at 1130, the indicated modulation rate may be obtained. At 1144,the indicated modulation rate 1130 may further be used in conjunctionwith an FM HRW to analyze and remove the oscillator peak to obtain thehigh resolution frequency domain samples 1132, which may further be usedfor convolution analysis 1148.

In an example, a method for FM detection may be elaborated. The complexspectral phase evolution methods may be extended so that they may beapplied to signals that are more complicated than the short-time stablesinusoids that were introduced earlier. In this example, a variation onthe CSPE may be introduced that may be applied to signals that may besweeping through a range of frequencies, and may determine with goodaccuracy the key underlying parameters that may define the sweepingfrequency.

An exemplary way to define a linear swept sine signal in the continuouscase may be as follows:

${x(t)} = {\sin\left( {2{\pi\left( {{f_{0}t} + {\frac{\delta}{2}t^{2}} + \varphi_{0}} \right)}} \right)}$

where f₀ may be the root frequency, δ/2 may be the frequency modulationrate and φ₀ may be the initial phase of the signal. In the case wherethe signal may be discretely sampled, a convenient form of the sweptsine signal may be:

$\overset{->}{x} = {\exp\left( {\frac{2\pi}{N}\left\{ {{\left\lbrack {{0\text{:}N} - 1} \right\rbrack f_{0}} + {\frac{\delta}{2}\left\lbrack {{0\text{:}N} - 1} \right\rbrack} + {\frac{\delta}{2}\left( {\left\lbrack {{0\text{:}N} - 1} \right\rbrack \odot \left\lbrack {{0\text{:}N} - 1} \right\rbrack} \right)}} \right\}} \right)}$

where [0:N−1] may be defined as to mean a vector of samples labeled 0,1, 2, . . . , N−1, and ([0:N−1]⊙[0:N−1]) may be the Hadamard/Schurproduct of the sample vector with itself. For the purpose of discussion,the Hadamard/Schur product of a vector with itself may hereinafter beabbreviated as [0:N−1]̂² in the following description. The operator ⊙ maybe defined to be the Hadamard/Schur product hereinafter. Withoutdeviating from the spirit and scope of this disclosure, the first twoterms in the curly braces may be combined as

${\left( {f_{0} + \frac{\delta}{2}} \right)\left\lbrack {{0\text{:}N} - 1} \right\rbrack},$

but it may also be convenient to write it in the uncombined form. Thenotation above may indicate a complex exponential form of the sinusoid(sometimes called the “analytic signal” by those skilled in the art),but one can easily convert back to the sine or cosine form by taking thereal or imaginary part of the complex exponential. In an example, thevector {right arrow over (x)} may represents a (complexified) sample ofN points from the swept sine signal, and a subscript may be added toindicate the last sample included in the vector, such that in anexample, the notation {right arrow over (x)}→{right arrow over(x)}_(N−1) may be used to represent that this vector of samples ends atsample N−1 (but it is implied that N total samples are included in thevector). Consequently, using this notation, the next possible group of Nsamples may be represented as depicted below:

${\overset{->}{x}}_{N} = {\exp\left( {\frac{2\pi}{N}\left\{ {{\left\lbrack {1\text{:}N} \right\rbrack f_{0}} + {\frac{\delta}{2}\left\lbrack {1\text{:}N} \right\rbrack} + {\frac{\delta}{2}\left( \left\lbrack {1\text{:}N} \right\rbrack^{\bigwedge 2} \right)}} \right\}} \right)}$

In keeping with the spirit and scope of the CSPE methods discussed inthe underlying description, the evolution of the signal from one groupof N samples to a later group of N samples may be analyzed. In anexample, this may be achieved by defining an evolution operator that mayadvance the signal so as to define Γ₁:{right arrow over(x)}_(N−1)→{right arrow over (x)}_(N) to be a one-sample evolutionoperator (applying it multiple times may advance the signal by more thanone sample):

${\overset{->}{\Gamma}}_{1} = {\exp\left( {\; \frac{2\pi}{N}\left\{ {{\overset{->}{f}}_{0} + {\delta \left\lbrack {1\text{:}N} \right\rbrack}} \right\}} \right)}$

whereby {right arrow over (f)}₀ may represent vector of length N whereeach entry may be the value f₀. Then by combining and refactoring it maybe observed that {right arrow over (Γ)}₁⊙{right arrow over(x)}_(N−1)={right arrow over (x)}_(N). This may be seen by the followingrearrangement of the vector terms in the exponent:

${{\overset{->}{f}}_{0} + {\delta \left\lbrack {1\text{:}N} \right\rbrack} + {\left\lbrack {{0\text{:}N} - 1} \right\rbrack f_{0}} + {\frac{\delta}{2}\left\lbrack {{0\text{:}N} - 1} \right\rbrack} + {\frac{\delta}{2}\left( \left\lbrack {{0\text{:}N} - 1} \right\rbrack^{\bigwedge 2} \right)}} = {{{\left\lbrack {1\text{:}N} \right\rbrack f_{0}} + {\frac{\delta}{2}\left\lbrack {1\text{:}N} \right\rbrack} + {\frac{\delta}{2}\left\lbrack {1\text{:}N} \right\rbrack} + {\frac{\delta}{2}\left\lbrack {{0\text{:}N} - 1} \right\rbrack} + {\frac{\delta}{2}\left( \left\lbrack {{0\text{:}N} - 1} \right\rbrack^{\bigwedge 2} \right)}} = {{{\left\lbrack {1\text{:}N} \right\rbrack f_{0}} + {\frac{\delta}{2}\left\lbrack {1\text{:}N} \right\rbrack} + {\frac{\delta}{2}\left\{ {\left\lbrack {1\text{:}N} \right\rbrack + \left\lbrack {{0\text{:}N} - 1} \right\rbrack + \left\lbrack {{0\text{:}N} - 1} \right\rbrack^{\bigwedge 2}} \right\}}} = {{\left\lbrack {1\text{:}N} \right\rbrack f_{0}} + {\frac{\delta}{2}\left\lbrack {1\text{:}N} \right\rbrack} + {\frac{\delta}{2}\left\lbrack {1\text{:}N} \right\rbrack}^{\bigwedge 2}}}}$

where the last step may follow from the general term:n+(n−1)+(n−1)²=2n−1+(n²−2n+1)=n².

${{\overset{->}{\Gamma}}_{1} \odot {\overset{->}{x}}_{N - 1}} = {\exp\left( {{\; \frac{2\pi}{N}\left\{ {{\left\lbrack {1\text{:}N} \right\rbrack f_{0}} + {\frac{\delta}{2}\left\lbrack {1\text{:}N} \right\rbrack} + {\frac{\delta}{2}\left\lbrack {1\text{:}N} \right\rbrack}^{\bigwedge 2}} \right\}} = {\overset{->}{x}}_{N}} \right.}$

In an example, the ability to specify the evolution operator may beimportant since the basic premise of the CSPE methods may be to comparea time-advanced (or, in some applications, space-advanced) snapshot of asignal with the original snapshot of the signal and then to isolateterms that may reveal the underlying parameters that may be used in amathematical reconstruction of the signal. As has been previouslydiscussed, the “frequency” f₀ may be held at the first instant in thegroup of samples, and it may be more convenient to reformulate theproblem so that the modulation may be considered relative to theinstantaneous “frequency” at the center of the window of N samples. Thequotes have been placed around “frequency” since it may be more accurateto consider f₀ to be the period of the signal, since a sinusoidal signalof the form

$x = {\sin \left( {{\frac{2\pi}{N}\left\lbrack {{0\text{:}N} - 1} \right\rbrack}f_{0}} \right)}$

may go through exactly f₀ periods in the N samples; however, it may becommon to call f₀ the frequency and one skilled in the art may be ableto determine the precise meaning based on the context of the usage. Inthis example, the modulation may be sweeping away from the initialfrequency f₀ and one may view this as setting the initial instantaneousfrequency in a group of N samples as being f₀. It may be possible toreformulate the modulation problem so that the modulation may be viewedas a modulation about an instantaneous frequency that may occur at thecenter of a group of N samples. This centered formulation may beconvenient and so it may be discussed further.

In an example, the creation of a modulating signal may begin with astable sinusoid, and Q periods over N samples may be taken such that:

{right arrow over (x)}=exp(i2π[0:N]Q/N)

Further, a (centered) Frequency Modulation Creation Operator (FMCO) maybe defined as:

${FMCO} = {\exp \left( {\frac{2\pi}{N}{{\frac{\delta}{2}\left\lbrack {{- \frac{N}{2}}\text{:}\frac{N}{2}} \right\rbrack}\bigwedge^{2}}} \right)}$

and when the FMCO may be applied to the sampled sinusoid x, the resultmay be a modulating signal, {right arrow over (y)} (here defined withN+1 points that will be used to study the signal evolution):

$\overset{\rightarrow}{y} = {{{\exp \left( {\frac{2\pi}{N}{\frac{\delta}{2}\left\lbrack {{- \frac{N}{2}}\text{:}\frac{N}{2}} \right\rbrack}^{2}} \right)} \odot {\exp \left( {\; 2{\pi \left\lbrack {0\text{:}N} \right\rbrack}{Q/N}} \right)}} = {\exp \left( {\frac{2\pi}{N}\left\{ {{\left\lbrack {0\text{:}N} \right\rbrack Q} + {{\frac{\delta}{2}\left\lbrack {{- \frac{N}{2}}\text{:}\frac{N}{2}} \right\rbrack}\bigwedge^{2}}} \right\}} \right)}}$

where y may be a linearly modulating signal, with an instantaneouscenter frequency corresponding to Q periods in an N point sample window.

In the exemplary embodiment discussed above, the linear frequencymodulation may be created in such a way that if δ=1/N, then the signalmay exhibit an increase of 1 period in every sequential non-overlappingN-point sample window. Thus, while it may be recognized that thefrequency may be increasing in a continuous and linear fashion, thedefined equation structure may lead to a signal with Q full oscillationsin the first N samples (such that a full oscillation may be defined tobe a passage through a full 2π interval), and in the next N samples, thesignal may exhibit Q+1 full oscillations, and in the next N samples thesignal may exhibit Q+2 full oscillations, and the like.

In an example, if the modulation parameter δ=2/N, then the formulationabove may give an increase of 2 periods in every subsequent window of Nsamples (non-overlapping). In an alternate example, if the windows areoverlapped by 50%, it may give an increase of 1 period in eachsubsequent 50% overlapping window, so if the signal exhibits Q fulloscillations over samples 1 to N, then for a 50% overlapping window ofsamples N/2+1 to N/2+N, the signal may exhibit Q+1 full oscillations andfor the next 50% overlapping window of samples N+1 to 2N the signal mayexhibit Q+2 full oscillations.

In a similar example, if the modulation parameter may be taken to beδ=P/N, then the signal may exhibit an increase of P periods in everysubsequent window of N samples (non-overlapping). In this example, theformulation of the signal frequency may be related to the value of Q(periods) through the usual transformations between frequency andperiod. The signal may be defined so that the instantaneous frequency atthe center of an analysis window may be equal to the frequency that maycreate Q periods in the window. The modulations may be around thatcenter frequency. In order to develop a method similar to the CSPE forshort-time stable sinusoids and extend the method to modulatingfrequencies, it may be necessary to develop a Frequency ModulationPullback Operator (FMPO) that may operate on the time-advanced (or insome cases spatially-shifted) data in such a manner that the frequencytransform of the resulting signal from the Hadamard/Schur product of theFMPO and the time-advanced signal may be nothing more than a phaserotation from the transform of the first signal.

In an example, the FMPO may be defined as illustrated below:

${FMPO} = {\exp \left( {{\pm }\frac{2\pi}{N}{\delta \left\lbrack {{- \frac{N}{2}}\text{:}\frac{N}{2}} \right\rbrack}} \right)}$

In this example, the sign of the imaginary unit, i, may be chosen to bepositive or negative depending on whether an up pullback operation or adown pullback operation may be desired. The CSPE technique formodulating signals may then become

${CSPE} = {{{F^{*}\left( {\overset{\rightarrow}{y}}_{N - 1} \right)} \odot {F\left( {{FMPO} \odot {\overset{\rightarrow}{y}}_{N}} \right)}} = {{^{{2\pi} \cdot \frac{Q + {\delta/2}}{N}}{{F^{*}\left( {\overset{\rightarrow}{y}}_{N - 1} \right)} \odot {F\left( {{FMPO} \odot {\overset{\rightarrow}{y}}_{N}} \right)}}} = {^{{2\pi} \cdot \frac{Q + {\delta/2}}{N}}{{F\left( {\overset{\rightarrow}{y}}_{N - 1} \right)}}^{2}}}}$

The derivation of this result may come from the following formulationwhere the exponent in (FMPO⊙{right arrow over (y)}_(N)) may beconsidered and the factor

$\frac{2\pi}{N}$

may be ignored for the purpose of the derivation:

${{- {\delta \left\lbrack {{{- \frac{N}{2}}\text{:}\frac{N}{2}} - 1} \right\rbrack}} + {\left\lbrack {1\text{:}N} \right\rbrack Q} + {\frac{\delta}{2}\left\lbrack {{- \frac{N}{2}} + {1\text{:}\frac{N}{2}}} \right\rbrack}^{\bigwedge 2}} = {{{\left\lbrack {1\text{:}N} \right\rbrack Q} + \frac{\overset{\rightarrow}{\delta}}{2} + {\frac{\delta}{2}\left\lbrack {{{- \frac{N}{2}}\text{:}\frac{N}{2}} - 1} \right\rbrack}^{\bigwedge 2}} = {\left( {\overset{\rightarrow}{Q} + \frac{\overset{\rightarrow}{\delta}}{2}} \right) + {\left\lbrack {{0\text{:}N} - 1} \right\rbrack Q} + {\frac{\delta}{2}\left\lbrack {{{- \frac{N}{2}}\text{:}\frac{N}{2}} - 1} \right\rbrack}^{\bigwedge 2}}}$

where the transformation from the first to the second line above may beseen by considering the general term:

${{- {\delta \left( {\frac{N}{2} - 1} \right)}} + {\frac{\delta}{2}\left( \frac{N}{2} \right)^{2}}} = {{\frac{\delta}{2} + {\frac{\delta}{2}\left( {\left( \frac{N}{2} \right)^{2} - N + 1} \right)}} = {\frac{\delta}{2} + {\frac{\delta}{2}\left( {\frac{N}{2} - 1} \right)^{2}}}}$

In the example above, putting all the elements together may give theresult that

$\left( {{FMPO} \odot {\overset{\rightarrow}{y}}_{N}} \right) = {{\exp \left( {\frac{2\pi}{N}\left( {Q + \frac{\delta}{2}} \right)} \right)}{\overset{\rightarrow}{y}}_{N - 1}}$

and the result above for the modulating CSPE follows. Consequently, ifthe angle of the modulating CSPE may be calculated, and further may benormalized by multiplying by N/(2π), the result may be exactly

$Q + \frac{\delta}{2}$

and this result may be found in any frequency bin if a single modulatingsignal were present. In practice, other interfering signals may bepresent, but the result may still hold in the region around the spectralpeak associated with the modulating signal. As a result it may beconcluded that this calculation may have been rendered a local operationin the frequency domain, and this may make it much more robust. If oneskilled in the art were to isolate Q and δ then the modulating signalmay be recreated exactly using the modulating signal creation techniquesas described within the scope described above.

In an exemplary method of determining the correct value of δ,calculation of the modulating CSPE for a set of modulation rates andfrom the resulting calculations, extrapolation or interpolation to thecorrect value of δ may be performed. The extrapolation may be done bymeasuring the width of the remapped spectral peak after calculating theCSPE and modulated CSPE for a few values of the modulation rate. Thecorrect value of the modulation rate may then be used to produce aspectral peak of near-zero width, and since the width of the spectralpeak may vary approximately linearly with the modulation rate, one mayuse interpolation or extrapolation to estimate the value of themodulation rate that may produce the near-zero width peak. This may havethe benefit of allowing the calculation of independent modulationparameters (i.e. 6 values) for several different signal components thatmay be present and may have different modulation rates. A secondapproach may be to use an iterative scheme to converge upon the optimalmodulation rate. In either case, the desired result may be detected bythe presence of a delta function-like spectrum that results from takingthe power in every frequency bin and re-plotting it at the locallymeasured value of

$Q + {\frac{\delta}{2}.}$

This may help to reduce the spectrum to a delta function when thecorrect value of 6 is used. If a number of different modulating signalsmay be present, then if a signal associated with a spectral peak p_(k)may have a modulation rate δ_(k), and then if the modulating CSPE may becalculated with δreplaced by δ_(k), then the resulting spectrum locallyaround peak p_(k) may be like a delta-function. Hence, either throughextrapolation/interpolation, or through iteration, it may be possible toisolate the central frequency values (corresponding to Q) or themodulation rates for linearly modulating signals.

In an example, a method for combined AM/FM detection may be defined. Themethods for amplitude and frequency modulation detection discussed thusfar may be used to detect either frequency modulation or amplitudemodulation, but not both. There may be several methods of integratingthese techniques into a coherent framework, including various decisiontrees, with and without mixed AM/FM detection, and tracker-assistedmodulation detection. An exemplary decision tree may be discussed in thefollowing description.

In an exemplary signal processing method, frequency modulation andamplitude modulation may be indistinguishable or intermixed. The methodmay include mapping the modulation into the complex plane, so thatradial changes may be considered as amplitude modulation, angularchanges may be considered as frequency modulation, and a co-variancematrix may be output into a tracking method. The tracker may then usethe information calculated over time to determine which portion of themodulation is better or more effectively characterized as amplitudemodulation and which portion is better or more effectively characterizedas frequency modulation.

FIG. 12 illustrates an example of a method using a decision tree 1200that may be used to combine AM/FM detection. The decision tree method1200 may include at 1204, using a CPSE twist to identify a source ofmodulation for a CPSE frequency spectrum 1202. The method 1200 mayfurther include using the modulation estimation 1208 obtained from theidentification step 1204 to perform one of the three exemplary processesillustrated in the FIG. 12. In a first example, the modulationestimation 1208 may provide a signal dominated by AM 1210. In a secondexample, the modulation estimation 1208 may provide a signal with mixedAM/FM 1212, while in a third example, the modulation estimation 1208 mayprovide a signal dominated by FM 1214. The decision tree method 1200 maythen include at 1218 and 1220, performing window type tests on thesignal dominated by AM 1210 and the signal with mixed AM/FM 1212respectively. The method 1200 may also include in an example, at 1222,interpolating an FM rate for the signal dominated by FM 1214. The method1200 may then include at 1224 and/or 1228 selecting applicable windowsfrom the windows obtained after performing window type tests 1218-1220.In an example, the method may include at 1230 selecting a correct FMwindow after the interpolation performed at 1222.

The method 1200 may further include, at 1232-1234 obtaining a windowsub-set and/or at 1238 obtaining a nearest FM window based on the stepsperformed at 1224-1230. The method may further include, at 1240, that isto say at 1240 a and at 1240 b, interpolating window parameters for thewindow subsets 1232-1234, to obtain at 1242, the estimated windowparameters, such as the estimated window parameter 1242 a and theestimated window parameter 1242 b. The method 1200 may further include,at 1244 performing convolution analysis on the estimated windowparameters 1242. Further, the method 1200 may include, at 1248,identifying residual error 1248 for the results of convolution analysis1244 and/or for the nearest FM window 1238. Based on the error, themethod 1200 may include, at 1250, choosing the best fit window,modulation, amplitude and phase and provide the results of selection asthe estimated signal parameters 1252. In its simplest form, the method1200 may be reiterated as including the steps of calculating thefit-error for a non-modulating sinusoid, calculating the fit-error forvarious AM HRW, calculating the fit-error for the closest available FMHRW, and choosing the HRW and fit parameters that may yield the lowestresidual error.

While the above steps may detect modulation effects more effectivelythan conventional FFT-based analysis, a more sophisticated decision treemay be used that may allow for the detection of both AM and FMsimultaneously. In an example, the sophisticated decision tree mayinclude calculating the apparent frequency modulation using any of theone or more FM detection methods discussed previously. The frequencymodulation may hereinafter by refer herein to as the ‘IndicatedModulation Rate’. The sophisticated decision tree may further includedetermining if the Indicated Modulation Rate is out of bounds or closeto zero. If the Indicated Modulation Rate is out of bounds or close tozero, the modulation may be dominated by amplitude effects. Thus,amplitude modulation windows may be used for an analysis of themodulation rate using any of the plurality of AM detection methodsdiscussed previously.

In an example of the sophisticated decision tree, if the indicatedmodulation rate is within certain ranges, the signal may be affected byboth amplitude and frequency modulation. Thus, the sophisticate decisiontree may include selecting a set of AM/FM HRWs. In an example, theamplitude modulation may skew the results of the CSPE flatnesscalculation. For example, a sinusoid with a positive frequencymodulation of 2 periods per window may create an Indicated ModulationRate of 2.0, but if the same sinusoid is also increasing in amplitude,it may create an Indicated Modulation Rate of 2.18. These effects may bepre-determined, and a calibration table may need to be created.

In accordance with one or more examples, there may be circumstanceswhere amplitude modulation and frequency modulation may beindistinguishable or intermixed. In these cases, knowledge of thebehavior of that oscillator in previous sample windows may be used toidentify the true modulation. For example, if the oscillator peakbelongs with a tracklet of data that may have been falling in frequency,it may be likely that the frequency may continue falling. In someexamples, the peak detection process may be aware of the state of thetracker so that it may make such inferences. In other examples, the peakdetection process may output ambiguous information that may be finalizedby the tracker. Further, in some examples, the peak detector may use thetrack information to utilize fewer steps in an AM/FM detection decisiontree, starting with the most likely AM/FM combinations.

In accordance with one or more examples, the multi-channelsuper-resolution method discussed previously may have as an output, aset of parameters describing individual oscillator components, and theirrelationship to each channel. In the set of parameters, each parametermay contain information that may be required to accurately reconstructthe oscillator with the use of such as the Unified Domain Re-synthesismethods. In a preferred example, that information may generally containfrequency, amplitude, Unified Domain sigma, amplitude modulation,frequency modulation, and the phase of the oscillator in each channel aswell as any appropriate amplitude or frequency modulation parametersthat may apply. The Unified Domain Sigma represents the portion of thesignal that may be derived from each channel.

In accordance with exemplary and non-limiting embodiments, AM and FMmodulation may be detected in a short duration window to achievesuper-resolution for AM and FM characteristics. In other embodiments,one or more frequency modulation pullback operators as described hereinmay be applied to at least one set of sample data.

In accordance with exemplary and non-limiting embodiments, frequencymodulation in a sample window may be detected. A plurality of frequencymodulation pullback operators (FMPOs) may then be applied to at least aset of sample data with the results subjected to one or more of aninterpolation, a linear interpolation, an extrapolation and an iterationto provide an improved estimate of an actual modulation rate. In someexamples, the plurality of FMPOs may include at least two of an “up,” a“down” and a “neutral” operation.

In a modification of the CSPE described above, in accordance withcertain exemplary embodiments, a sample window, such as a “hammingwindow” or other standard windowing function or “tapers” may be usedbut, when dealing with an FM input signal, there may be inserted anothervector (the FMPO—frequency modulation pullback operator) that mayinformally be hereinafter be referred to as a “twist vector”. Typically,the FM signal may be moving in frequency as one receives the time-laggedversion. In addition, AM signals tend to evolve in time like a rigidrotator; however, unlike the case of the effect of the analysis windowused in the standard CSPE, for the AM modulation case the AM window maybe a part of the data. Thus, when detecting the rotation of the “rigidrotator” of the AM window, one must allow for the shift in the AM windowin the time-lagged version. For the frequency modulation case,application of the FMPO may turn the FM frequency back to something thatmay evolve like a rigid rotator, from which can be detected thefrequency and the angular rotation. One can then derive the FMmodulation and the reference “root” or “anchor” point for the frequencymodulation representation.

In accordance with other exemplary and non-limiting embodiments, AM andFM modulation may be detected in a short time window to achievesuper-resolution for AM and FM time windows. Complex spectral phaseevolution (CSPE) may then be performed on the frequency domain data toestimate component frequencies at a resolution and/or an accuracy thatmay be greater than the fundamental transform resolution.

In an example, the amplitude effect of the AM signal may be different onthe first snapshot or window of samples versus the lagged window ofsamples. Accounting for how the amplitude effect changes may allowderivation of the underlying signal from the CSPE. Conversely, asdescribed above, with FM signals the FMPO may be used to derive theunderlying FM behavior. With AM signals, the evolution of the amplitudeeffect may be considered.

In an example, creating the amplitude modulation windows may includetaking a stable signal, applying an amplitude effect, and putting theresult into a high-resolution FFT. By subsequently considering a varietyof amplitude modulation effects, including but not limited to, AMeffects where the amplitude envelope slopes upward, AM effects where theamplitude envelope slopes downward, AM effects where the amplitudeenvelope starts or stops at an arbitrary point in the data sample, AMeffects where the amplitude envelope may have a combination of effectsthat may include sloping upward or downward or leveling off ortransitioning smoothly from one AM envelope state to another, and anycombination of these states, one may derive a series of high resolutionoscillator peaks from which may be determined which one of the appliedeffects may fit best to a given component of the frequency spectrum.

In some examples, a plurality of amplitude effects may be pre-computedand multiplied by the analysis window. These amplitude effects may thenbe converted to the frequency domain via a high-resolution FFT or othertransform and may be compared to the spectral peaks detected in thesignal to determine the amplitude modulation effect that may beassociated with the observed structure of the spectral peak. Examples ofthese AM effects may include, but are not limited to AM effects wherethe amplitude envelope slopes upward, AM effects where the amplitudeenvelope slopes downward, AM effects where the amplitude envelope startsor stops at an arbitrary point in the data sample, AM effects where theamplitude envelope has a combination of effects that include slopingupward or downward or leveling off or transitioning smoothly from one AMenvelope state to another, and any combination of these states.

In accordance with another exemplary and non-limiting embodiment, aplurality of amplitude effects, such as commonly known amplitude effectswithin a library of possible known amplitude effects, may be applied inturn to a reference signal that may then have a high-resolutiontransform/FFT applied to it. Within a complex spectral phase evolutionrepresentation of a signal, an oscillator peak of an underlying signalthat was modified by some amplitude effect may be analyzed, in order todetermine which of the set of possible amplitude effects, when appliedto the underlying signal/stable oscillator, results in the best fit toan actual oscillator peak of the underlying signal. Information knownabout the context of the signal, such as whether it is speech versusartificial sound, may be used to provide further assistance in thedetermination of what amplitude effect is likely the best representationof the change in amplitude over time of the underlying signal.

In accordance with exemplary and non-limiting embodiments, transientsignal elements (onset and stop) may be treated as AM signals in a superresolution signal processing method. In addition to onset and stop,other transient signal elements including, but not limited to, risingup, rising down, or a generic envelope may be so treated.Mathematically, a sharp noise that occurs over a short time may beconsidered transient on some scale, where the short time duration of thenoise is shorter than the sample window time. A short enough event maytend to have a very sharp envelope, and that envelope itself may be likean AM effect. As a result, in some examples, the present methodology maybe configured to handle transient signal elements as manifestingthemselves as AM signal effects.

In accordance with one or more examples, the signal componenttracker/aggregator/selector/separator 214 as illustrated in FIG. 2, andhereinafter referred to as the signal component tracker 214, may bedescribed. The function of the signal component tracker 214 inaccordance with one or more examples may be to group and extractoscillator peaks for subsequent re-synthesis or output into one of theoutput formats.

FIG. 13 illustrates an example of a method 1300 performed by the signalcomponent tracker. The signal component tracker may include an inputmodule that may be configured to receive sets of signal oscillator peaksconstructed by either the Single Channel Super-Resolution module or theUnified Domain Super-Resolution Module. Signal oscillator peaks may begiven structure and organization by tracking methods. The oscillatorpeaks may be organized into sets of oscillator peaks, where each memberof a set may be determined to be caused by the same oscillator(“Tracklets”). The method 1300 may include, at 1304, using the inputprovided by the input module 1302 to associate ‘tracklets’ using anassociation criteria that may include frequency, magnitude, UnifiedDomain Sigma, and other attributes to identify peaks from the sameoscillator. The method 1300 may further include, at 1308, associatingthe ‘tracklets’ using their harmonic relationships to identify sets oftracklets created by the same source to group these tracklets into oneor more “Coherent Groups”. The grouping of tracklets may be performedusing any of a plurality of tracking algorithms known in the art.Further, the method 1300 may include, at 1310, selecting the coherentgroups and at 1312, separating the coherent groups to provide a filteredoscillator peak matrix 1314 for re-synthesis.

In an example, the tracking algorithms known in the art may include aMulti-Hypothesis Tracking (MHT) method. The method may includepreserving multiple possible data associations until data is receivedthat may confirm a correct association. In some other examples, thetracking algorithm may use other well-known algorithms to associatetracklet to oscillator peak, such as Greedy, Munkres, or JVC. In analgorithm, a Kalman Filter may be applied to predict the motion of atracklet in several dimensions including, but not limited to, frequencyand amplitude. Further, well-known grouping algorithms may also beapplied to the problem of identifying tracklets emanating from the samesource, such as Union Find. For example, a Track Fingerprintingalgorithm may be used, which works by identifying individual signalsources using the source's harmonic patterns.

Thus, in accordance with an exemplary and non-limiting embodiment, asignal processing method may include super-resolution analysis andgrouping of signals into frequency groups of tracklets, which formrepresentations of the time evolution of oscillators, and aggregatingthe data into coherent groups of tracklets via a grouping algorithm toidentify coherent groups of frequencies within a signal. The groupinginto tracklets may be performed using a tracking algorithm such asKalman Filter, greedy association or any other such algorithm as knownto those skilled in the art, to identify short term stable oscillatorsthat may come and go as a signal source evolves through time. The datamay be further divided into coherent groups of tracklets usingcombinations of well-known track grouping algorithms, such as, withoutlimitation, union find.

In some examples, the analysis may be used to aggregate signal elementsinto tracklets.

In some examples, partitioning may be used to aggregate signal elementsinto coherent groups.

In some examples AM and FM affects may be detected and/or correctedusing the measured evolution of frequency and amplitude of oscillatorpeaks contained in a tracklet.

In some examples, the evolution of frequency and amplitude in a trackletor a coherent group may be used to identify speech or non-speech. Forexample, speech tends to curve through frequency over time, whereasnon-speech is often flat in frequency as it evolves in time.

In some examples, a human may be presented with an interface to viewoscillator peaks, tracklets and coherent harmonic groups.

In some examples, a human may assist the system in determining whichoscillator peaks, tracklets and harmonic groups may be output oraccentuated.

In some examples, the system may learn from a human's choices aboutwhich oscillator peaks, tracklets and harmonic groups should be kept.

In some examples, the system may learn vocal patterns of an individualspeaker or signal source. These patterns may include harmonicseparation, rates of change of frequency and/or amplitude, oraggregations of any other data that may be contained in the oscillatorpeak.

In some examples, the oscillator peak detection, tracking and groupingprocess may be used for audio signals.

In some examples, the oscillator peak detection, tracking and groupingprocess may be used for any signal, including, but not limited to RADAR,SONAR, LIDAR, and sound/audio, video, and vibration sensors.

In some examples, the tracklets may be used to form coherent groups.That process of forming coherent groups is called “partitioning” in thetracking. For example, in sound sources, each tracklet may represent aharmonic. A tracklet may typically move through frequency, time anddirection of arrival (that is, related to the sigma of the unifieddomain) and may vary in amplitude along the tracklet. These shapesrepresent AM or FM effects that are detected. “Grouping” as used hereinmay refer to attempts to find the harmonics amongst tracklets that maybe moving together and may be from one coherent source. When viewedvisually, as described below, sweeping curving lines in a time-frequencyrepresentation of a sound signal may be typically indicative of speech,while flat horizontal lines may be often indicative of artificialnoises, such as car alarm sounds. As a result, naturally occurringversus artificial sounds can be separated by, for example, shape andtype in the partitioning. Separation can also be based on any otherparameters that may be calculated in the analysis process. As a result,an automated algorithm may be employed to eliminate artificial sounds orenhance artificial sounds as desired for a given application. Inaccordance with exemplary and non-limiting embodiments, a user interfacemay be provided for viewing a signal as a plurality of potentiallycoherent tracklets in order to edit the visual representation toidentify signals as belonging to a desired tracklet or coherent groupand scoring an element of the signal based on feedback from the editingof the visual element. In other embodiments, a sound signal, such asbased on speech from an individual may be introduced for identifyingparameters that may facilitate grouping of tracklets that correspond tosignals produced by the individual. In this way, one may performlearning on the patterns of a speaker such that there is derived acharacteristic feature set for that speaker. In another embodiment,speech originated by an individual may be identified based on“fingerprinting” of a source based on unified domain parameters that arecharacteristic of the known signature or “fingerprint” of thesource/individual. In other embodiments, the source signal may be any ofthe other types of signals discussed within the scope of thisdisclosure.

In addition to well-known tracking algorithms, the tracker may employnew algorithms to improve output quality. For example, a Phaseprediction algorithm may be used to predict the likelihood that twopeaks emanate from the same sound source. In an example, peak correctionmay be performed using Phase Prediction.

FIG. 14 illustrates an example a method 1400 performed by the signalcomponent tracker 214 that may use phase prediction. The method 1400 mayuse phase prediction as a criteria for associating ‘tracklets’ inaddition to the association criteria discussed on in FIG. 13. In someexamples, the output of frequency-phase prediction may cause the trackerto re-calculate the parameters associated with an oscillator peak. Insome examples, peak correction may be performed using proximity. Forexample, the tracker may calculate that two oscillator peaks areinterfering, and may use the track state information to correct thatinterference. The oscillator peaks may then be selected for providing anoutput.

In an example, the methods of peak selection may include, but are notlimited to, evaluating the peak using parameters, such as Unified DomainSigma, Frequency, and Amplitude, evaluating the tracklet to which thepeak belongs using parameters, such as Unified Domain Sigma, Frequency,and Amplitude, evaluating the coherent group to which the peak maybelong using parameters, such as Unified Domain Sigma, Frequency, andAmplitude, evaluating whether the coherent group to which the peak maybelong matches a desired speaker using harmonic “fingerprinting,” usingfrequency-phase prediction to identify whether the tracklet appears tobe a ‘direct-path’ source, and may discount peak's parameters that mayfail to pass the peak prediction tests when evaluating a tracklet's or acoherent group's parameters, or estimating a distance by combining theUnified Domain Sigma with the phase information.

As previously discussed, a tracklet may be defined as a grouping ofoscillator peaks that may be determined to emanate from one sourceharmonic. In an example, a tracklet formation process may be performedon the basis of an input comprising a set of oscillator peaks extractedfrom a single sample window. Tracklets may then be formed using manydifferent well-known track association methods and algorithms. Thealgorithms may involve a method that may predict a tracklet forward, amethod that may compute a cost of association between a tracklet and anew piece of data (in this case, an oscillator peak), and a method thatmay choose an optimal set of assignments. By way of example, thetracklet prediction algorithm may include, but are not limited to,linear prediction, and Kalman Filter prediction. In some examples, thecost estimation algorithms may include, but are not limited to,statistical distance calculation, such as a Mahalanobis Distance, andsimple distance calculations, such as difference in frequency andamplitude. Further, in an example, assignment algorithms may include,but are not limited, to Greedy Association, Munkres Association, and JVCassociation.

The output of the tracklet formation process may include a set ofoscillator peaks that have been formed into tracklets.

As previously discussed, in some examples it may be desirable to formcoherent groups. A coherent group may be a set of tracklets that mayhave been determined to be produced by the same sound source.

In accordance with exemplary and non-limiting embodiments, a signalprocessing method may include super-resolution analysis, assigningsignal elements into frequency tracklets from snapshots in time (whereinthe snapshots in time may indicate using a sample window of datastarting at an initial time and ending at a final time, multiplying itby an analysis window, and converting it to the frequency domain), thatis to say, organizing the data into tracklets by a tracking algorithm toidentify frequency tracklets within a signal, and using at least one ofthe frequency, angle of arrival, amplitude, and slope of the amplitudeof a track in order to assist in grouping tracklets into coherentgroups.

As previously discussed, in some examples it may be desirable to formcoherent groups. A coherent group may be a set of tracklets that havebeen determined to be produced by the same sound source. In an example,coherent groups may be formed by a process that may receive a set oftracklets as input. The set of tracklets may then be partitioned intodisjoint sets. There are several well-known algorithms for partitioningthe sets of tracklets into disjoint sets. For example, the Union Findalgorithm may be employed. For most of the algorithms, a cost functionmay need to be calculated to compute the likelihood that two trackletsare from the same source. These cost functions may use any of thefeatures stored in a Singlet representation of an oscillator peakincluding, but not limited to, frequency, amplitude, Unified DomainSigma, and phase. These cost functions may rely on knowledge of theharmonic structure of a speaker. As a result of performing the coherentgroup formation process, as an output, a set of tracklets that have beenformed into coherent groups may be generated.

In accordance with an exemplary and non-limiting embodiment, a unifieddomain directional estimate may be used with the outputted set oftracklets to identify a tracklet of interest.

In some embodiments it may be desirable to predict the evolution ofphase as an oscillator evolves through time and frequency. Knowledge ofpredicted phase may be used in several stages of processing, including,but not limited to the activities discussed in the above description,including scoring the likelihood that a peak should be associated with aparticular tracklet, detection and/or repair of results from interferingsignals, detection and/or repair of dropped out or missed signals,detection of direct-path versus non-direct-path signals, compressionalgorithms, and association of tracklets into coherent groups.

In an example, the basic model of the signal may be taken as theprojection into the real numbers of the general complex forms(t)=r(t)e^(iθ(t)). Further, it may be assumed for the example that overa short period of time the amplitude term may remain constant, that isto say, r(t)=r₀, then the rate of change of the signal may be related tothe rate of change of θ and this may then be related to theinstantaneous frequency. This may give

$\frac{s}{t} = {\; r_{0}\frac{\theta}{t}^{{\theta}{(t)}}}$

and since the rate of change of θ may include the instantaneousfrequency, this may give a way to relate the frequency and phase of asignal that may be evolving in time. In practice, the parameters may beestimated based on the measured data, and the high-resolution analysismay make it possible to make accurate estimates of the instantaneousfrequencies, and hence accurate predictions of future frequency andphase values.

The algorithm for frequency phase prediction may start with thedifferential equation relating phase and instantaneous frequency,

$\frac{\theta}{t} = {f(t)}$

and over a sufficiently short window of time, f(t) may be approximatedas a constant plus linear term (and for one skilled in the art theexpansion can be continued easily to higher order terms), givingf(t)=f₀+at.

The example may further include estimating the frequencies from thedata, and this may be done using the super-resolution analysis from theCSPE family of transforms. In an alternate example, the frequencyestimate for a given sample data window, say the i^(th) window, may beassumed to be most accurate at the center of the window—which mayhereinafter be referred to as f_(i). The best position for the frequencyestimate may be obtained by a variety of other methods, including butnot limited to, checking zero crossing rate, looking at residual errorafter fitting with the frequency estimate and the like. Thesuper-resolution frequency estimate for the j^(th) window may be givenby f_(j) and it may be assumed to be most accurate at the middle ofsample data window j (or again positioned by a variety of othermethods). Thus the super-resolution frequency estimates and the relatedtimes where the estimates are positioned may be obtained. If thedifferential equation may be solved assuming boundary conditions thatmay be consistent with the frequency estimates, the first orderapproximation may include

$a = \frac{f_{j} - f_{i}}{2\; T}$

where T may be the time between the two frequency estimates (and in apreferred embodiment may be taken to be the time shift between the twosample data windows). The net result may be that one may derive thephase as a function of time, giving

${\theta (t)} = {{2{\pi \left( {{f_{i}t} + {\frac{f_{j} - f_{i}}{2\; T}t^{2}}} \right)}} + \theta_{0}}$

where θ₀ may be the initial phase of the signal. The approach presentedhere may be shown to be highly accurate when used with thesuper-resolution frequency estimates.

In some examples, the phase of a tracklet may be predicted to adifferent time. The difference between that prediction and a newoscillator peak's measured phase may be used to score the likelihoodthat the new peak should be incorporated into the tracklet.

In some examples, if two tracklets may be determined to be interfering,the predicted frequency and phase of each tracklet may be used to repairthe interfering regions so that the combined signal power may bereassigned to the interfering tracklets.

In some examples, if a tracklet may be determined to be missing data ina given frame, an estimated oscillator peak may be inserted using apredicted frequency, amplitude, phase, modulation type, direction ofarrival, and any other characteristic that may be included in thetracked oscillator peaks.

In some examples, a difference between a tracklet's predicted phase andactual phase may be used to determine whether it travelled to the sensorin a direct path or via an indirect path.

In accordance with another exemplary and non-limiting embodiment, thefrequency/phase of signal representations may be predicted based onsuper-resolution, unified domain model of coherent signal elementswithin a signal, and a signal element may then be processed based on theprediction. For example a prediction of every other frame may be used,allowing skipping of the processing of the predicted frame. As a result,for example, it may only be necessary to process frames 1, 3, 5, 7 inorder to predict frames 2, 4, 6 and 8. In this example, a prediction offrame 2 may be performed and further an estimate of what frame 2 turnedout to be may be done, thus providing a measurement of accuracy. In thisway, it may be determined, for example, how closely did two spectralpeaks so created line up. If the alignment is above a certain tolerance,frame 2 may be recalculated to make sure that it may be within anacceptable and predetermined range of error. If the alignment is withinthe accepted tolerance, then no further prediction may be required.

As described above, in accordance with exemplary and non-limitingembodiments, one may predict the frequency/phase of signalrepresentations based on a super-resolution, unified domain model ofcoherent signal elements within a signal. The resulting models may beaccurate enough to allow for the prediction of the evolution of signalelements through frequency, amplitude, phase, and time. As a result, ifsome data may be missing, it may be possible to keep the signal elementtracks/tracklets going across gaps. In the instance of trackletintersection, predicted values may be utilized to determine the behaviorand direction of the underlying tracklets. In one embodiments,predictive interpolation of gaps in signals may be performed bothforward and backward to determine a consistent estimate of the missingor obscured data. In one embodiment this may be implemented on a cellphone network to ameliorate the effects of dropped packets.

In other examples, the frequency and/or phase of signal representationsmay be predicted based on a super-resolution, unified domain model ofcoherent signal elements within a signal and grouping a signal elementwith other elements based on the prediction. The measurements of thesignal are typically accurate enough to allow for prediction forward intime in a manner that is more accurate than random. Put simply, it maybe predicted that a tracklet goes somewhere, and then when anobservation regarding such a tracklet with those properties is made, itmay be derived that the observed phenomenon is in fact associated with asignal element encountered before.

In accordance with another exemplary and non-limiting embodiment, thefrequency/phase of signal representations may be predicted based onsuper-resolution, unified domain model of coherent signal elementswithin a signal, and a signal element may then be processed based on theprediction. For example a prediction of every other frame may be used,allowing skipping of processing of the predicted frame. As a result, forexample, it may only be necessary to process frames 1, 3, 5, 7 in orderto predict frames 2, 4, 6 and 8. In this example, a prediction of frame2 may be done and a quick estimate of what frame 2 turned out to be mayalso be performed, thus providing a measurement of accuracy. In thisway, one may determine, for example, how closely did two spectral peaksso created may line up. If the alignment may be above a certaintolerance, frame 2 may be recalculated to make sure that it may bewithin an acceptable and predetermined range of error. On the otherhand, if the alignment may be within the accepted tolerance, then nofurther prediction may be required.

In some examples, the frequency/phase of signal representations may bepredicted based on super-resolution, unified domain model of coherentsignal elements within a signal and may be used to provide data tocomplete an incomplete signal representation based on the prediction.

In some embodiments, frequency/phase of signal representations based onsuper-resolution, unified domain model of coherent signal elementswithin a signal may be predicted and used to process a signal elementbased on the prediction, wherein processing the signal element mayinclude using the prediction to facilitate compression of arepresentation of the signal. As above, by not having to process everyframe, the information retained in the mathematical representation canbe represented with far fewer bits than the original data (that is tosay it may be naturally compressed).

The oscillator peak detection stage may use information from the currentstate of the tracking and/or grouping stages to guide its processingdecisions. Techniques may further include reduction of interference bytrack aware fitting and prioritization of oscillator peak selection, asdescribed below.

In an exemplary technique, reduction of interference through track-awarefitting may be implemented. When two oscillator peaks are on nearly thesame frequency, they may interfere, and be indistinguishable. If twotracklets are detected to be on a trajectory that will intersect infrequency in a given frame, one may use predicted frequencies andamplitudes to create two oscillator peaks where the system may onlydetect a single peak that is the sum of both oscillators.

In another exemplary technique prioritization of oscillator peakselection may be done. Due to computational resource limitations, or adesire to optimize performance or battery life or a number of otherfeatures, the system may fit fewer oscillator peaks than it detects. Thesystem may use the tracklet and/or group state information to pick theoscillator peaks to fit. For example, in sound processing, if a speakerof interest may show a consistent harmonic separation, the system mayfirst attempt to find oscillator peaks that may fit the existingpattern. Similarly, psycho-acoustic measures of the importance of signalcomponents may be used to prioritize which oscillator peaks should beprocessed.

In another exemplary technique extraction of desired signals from noisyenvironments or enhancement of desired signals in noisy environments—thetracking and grouping algorithms may be used, along with any of themeasured parameters of the fitted data, to determine which tracklets orcoherent groups should be extracted from the noise, or enhanced over thenoise.

In accordance with an exemplary and non-limiting embodiment, anambiguity measure or certainty measure may be assigned to the trackletsby the tracker. This ambiguity measure may be used in a Kalman filter, aBayesian decision process, a scoring function or a similar processwhereby the certainty/ambiguity measure is used to determine whichtracklets or coherent groups should be extracted or enhanced. In yetother embodiments, the intersection of a plurality of tracklets may beidentified with prediction of tracklet direction used to assist in thehandling of intersection points. For example, in sound processing, whentracklets actually cross, one merged sound at one frequency may beobserved. In this example, the merged sound may be taken at anintersection point and may be assigned to each of the tracks so thatthey may be self-consistent.

In an example, the output may include a subset of the peaks that werereceived on input. In some cases, these may be modified, such as in thecase of frequency phase prediction correction.

A range of techniques may be used to identify relevant oscillator peaksand tracklets. In an example, a processor may receive a plurality ofoscillator peaks and may select one or more of the plurality ofoscillator peaks for re-synthesis.

In an alternate example, oscillator peaks may be scored to determinewhich are desired for output. Oscillator peaks may be scored using atleast one of time, frequency, phase, amplitude, and unified domaindirection of arrival.

In an alternate example, tracklets may be scored to determine which aredesired for output. Tracklets may be scored using at least one of time,frequency, phase, amplitude, unified domain direction of arrival, changein any of those characteristics, and predictability of change in any oneof those characteristics.

In an alternate example, coherent groups of tracklets may be scored todetermine which are desired for output. Coherent groups may be scoredusing at least one of time, frequency, phase, amplitude, unified domaindirection of arrival, and change in any of those characteristics, andpredictability of change in any one of those characteristics, andconformance to a known harmonic structure, such as a person's knownharmonic frequency patterns.

In an alternate example, a peak, tracklet, or coherent groups score maybe used to assign it for output, or eliminate it from output.

In an alternate example, a peak, tracklet, or coherent group's score maybe used to modify its amplitude in output, thereby reducing oramplifying its impact.

In an alternate example, a peak, tracklet, or coherent groups score maybe communicated to another system, such as a speech recognizer, to aidit in its estimation process.

In an alternate example, original signal may be combined withreconstructed signal for output. Either the original or reconstructedsignal may be diminished or amplified before combination.

In an alternate example, elements such as background noise, otherinterfering signals, or any other signal with undesirablecharacteristics may be rejected or diminished.

In an alternate example, elements such as background noise, secondary orother interfering signals may be revealed by removing a primary signalthat may be obscuring the background.

In some examples the signal channel re-synthesis module 220, asillustrated in FIG. 2 may be used in accordance with one or moreexamples to create a frequency domain representation of the targetedoscillator peaks in a single output channel. In an example, the selectedoscillator peaks may be converted back to frequency or time-domainsignal using single channel re-synthesis. For some applications, suchoscillator peaks may be the output of the system.

The input to single channel re-synthesis module 220 may be a set ofoscillator peaks containing the parameters that may be used to createfrequency domain representations of those oscillator peaks in a singlechannel. In one or more examples, the oscillator peaks may generallycontain any of the parameters, including but not limited to, frequency,amplitude and phase. Further, the parameters of the analysis window usedwith Sample Window (A) and Sample Window (B) may be those determined inthe single channel pre-processor.

In some examples, the single channel re-synthesis module 220 may beconfigured to perform a method to use each oscillator peak received atthe input to calculate a frequency domain data projection. The methodmay include creating a normalized frequency domain representation of theoscillator by sampling the high resolution frequency domain version ofthe analysis window used to taper Sample Window (A) and Sample Window(B) in the Single Channel Pre-Processor. Multiply the normalizedfrequency domain representation of this oscillator by the oscillatorpeak's amplitude and phase. The method may further include, summing thespectrum created previously, once the frequency domain datacorresponding to the oscillator peaks has been calculated. In anexample, if time-domain data may be required, an inverse-FFT (iFFT) maybe performed that may convert the frequency output to the time domain.

In some examples, some amount of background signal may be required toprovide desirable characteristics in the output. An advantage of themethods used herein is that the phase of the output signal may bepreserved with high accuracy. As a result, the phase of the samples inthe original signal may match the phase in the extracted andre-synthesized signal. In some circumstances, addition of the backgroundsignal may yield a result that has desired characteristics. This may beachieved by a variety of techniques, including mixing back in theoriginal signal or an attenuated or amplified version of the originalsignal. In some examples, it may be desirable to use the singletrepresentation of the original signal, such as a singlet representationin a compressed form, so that the original signal may be reconstitutedbefore remixing with the extracted signal.

Based on the method performed by the single channel re-synthesis module220, a set of frequency domain or time-domain data that accuratelyrepresents the portions of the original signal corresponding to theselected set of oscillator peaks may be obtained as the output from thesingle channel re-synthesis module 220.

In some examples the multi-channel re-synthesis module 222, asillustrated in FIG. 2 may be used in accordance with one or moreexamples to create a frequency domain representation of the targetedpeaks in a multi-channel output. The multi-channel re-synthesis module222 may be configured to convert selected oscillator peaks back tofrequency or time-domain signals. In some examples, such oscillatorpeaks may be the output of the system.

The multi-channel re-synthesis module 222 may be configured to receiveas an input, a set of oscillator peaks containing the parameters used tocreate frequency domain representations of those oscillator peaks inmultiple channels, and the parameters of the analysis window used withSample Window (A) and Sample Window (B) in the multi-channelpre-processor 210. In an example, the oscillator peaks may contain:frequency, amplitude, Unified Domain Sigma, and the phase of theoscillator peak in each channel.

The multi-channel re-synthesis module 222 may be configured to perform amethod for each oscillator peak to calculate its frequency domain dataprojection for each channel. The method may include calculating theamplitude for that channel for that peak using the Unified Domain Sigmaand the input amplitude. The method may further include creating anormalized frequency domain representation of the oscillator by samplingthe high resolution frequency domain version of the analysis window usedwith Sample Window (A) and Sample Window (B) in the Single ChannelPre-Processor. Multiply the normalized frequency domain representationof this oscillator by the amplitude calculated in step 1 and theoscillator peak's phase for that channel, as received in the input. Themethod may further include summing the spectrum created in the previousstep once the frequency domain data corresponding to the oscillatorpeaks has been calculated. In an example, the frequency domain forchannel X may be the sum of all the calculated frequency domain spectrumfor channel X for all oscillator peaks. If time-domain data may berequired, an inverse-FFT (iFFT) may be performed to convert thefrequency output to the time domain.

In an example, a re-synthesized signal may be built in a signalprocessing model, using a convolutional model and using distinct methodsto build each of a plurality of signal elements or characteristics,including stable frequency signals, FM peaks, and AM peaks. Oscillatorpeak parameters, including frequency, amplitude, frequency modulationand amplitude modulation may be re-calculated to predict the parametersthat may exist in a different window position (such as slightly later intime) or window length as follows:

In an example a different window length may be chosen for resynthesisthan may have been used for oscillator peak detection. This may enablethe system to use a more optimal window length for re-synthesis than mayhave been used for oscillator peak detection.

In an example, the length of the sample window may be adjusted asnecessary. Under certain condition, it may be optimal to use a samplewindow of a different length. This may be done because the parameters ofthe detected oscillator peaks may be adjusted for comparison. Forexample, during periods of intense frequency modulation it may beadvantageous to sample more frequently.

In some examples, frames may be shifted within re-synthesis. Forexample, a first snapshot of a signal may be taken using for example,from 0 to 1024 data points. A next might start with data point 512 andcontinue to data point 1536, shifting one-half of the window length. Ifchanges on a smaller scale may be desired, shifting by fewer data pointsmay be desired (such as shifting by 256), then shift again, in whichcase each signal element is covered more closely. At re-synthesis, justthe middle segments may be used (256 to 768 and 512 to 1024) andpredicting forward within a frame may be done to make smaller frames.For example, if while processing a video input signal and sending videofrom wide screen format to old fashioned television, operation may bechanged on the fly so that the center of the TV image may be reproducedand the edges may be eliminated without decreasing quality and withoutbeing required to undertake complicated manipulation of the frames.

Based on the method performed by the multi-channel re-synthesis module222, a window of data in the frequency or time domain that mayaccurately represent the portions of the frequency domain from theoriginal signal corresponding to the selected peaks may be obtained asthe output from the multi-channel re-synthesis module 222.

The signal separation (SS) technology described herein may be applied toany system that may send or capture signals through a collectionmechanism (such as including a microphone, a camera, radio receiver, avideo camera, a transducer, or other receiver) for either transmission,storage, analysis or manipulation. The signal may subsequently be (butnot limited to): transmitted between receivers (e.g. RF transmission);delivered in an audio format, such as for transmission of a voice call,delivered in an image or video format such as transmission of a photo orvideo, depicted in a text-format such as converted from speech to text,or interpreted and rendered as an image such as a radar display orultrasound.

In some embodiments, SS technology may be introduced into one or moreprocesses and/or systems that involve digital signal processing. Digitalsignal processing is generally defined as the mathematical manipulationof an informational signal to modify or improve it, and may becharacterized by the representation of discrete time, discretefrequency, or other discrete domain signals by a sequence of numbers orsymbols and the processing of these signals. Sample digital signalprocessing fields where SS technology may deliver benefit may includebut are not limited to, audio processing and compression, speechprocessing and recognition, RF transmission, biometric analysis, sonarand radar, sensor array, ultrasonic testing, spectral estimation,statistical analysis, digital image, digital and cellularcommunications, control systems, biomedical, medical imaging, andseismic data. Digital signal processing may be applied to measure,filter and/or compress continuous real-world analog signals. The processmay typically begin by converting the signal from an analog to a digitalform, by sampling and then digitizing it using an analog-to-digitalconverter (ADC), which may turn the analog signal into a digital streamof numbers. Typically, after analysis and transmission, the requiredoutput signal may be another analog output signal, which requires adigital-to-analog converter (DAC).

In some embodiments, the SS technology may be implemented on, but notlimited to, one or more of the following: general purpose computers andGPUs; specialized single and multi-core processors (such as DigitalSignal Processors); purpose-built hardware such as application-specificintegrated circuit (ASICs); field-programmable gate arrays (FPGAs);digital signal controllers; and stream processors. In addition, the SStechnology described herein may be implemented as firmware, embeddedsoftware, a software platform, a standalone software application, and/ora network or cloud-based application/service. Such implementations maybe applied, but not limited to: computers; cellular phones or smartphones; tablets; or other communications; audio, video, sensor, radar,sonar or medical-imaging devices or systems; or any other system ordevice whereby digital signal processing may improve performance orgeneral usefulness.

In some embodiments, the signal separation technology described hereinmay be utilized in Radar-based object detection and tracking systemsthat rely on radio waves as a method to determine the range, altitude,direction, speed or other characteristics of objects. The radar systemsmay incorporate a process of transmitting pulses of radio waves (ormicrowaves), which are reflected off any object in their path, andsubsequently return a portion of the wave's energy to a receiver. Someexemplary uses of radar may include, but are not limited to: generalimaging, air defense and anti-missile systems, air traffic control,marine systems to locate terrain, vessels and other marine-based pointsof interest, aircraft anti-collision systems, ocean surveillancesystems, outer space surveillance and rendezvous systems, meteorologicaltracking and monitoring, altimetry and flight control systems, guidedmissile target locating systems, terrain mapping, detection and locationsystems, oil and gas discovery and drilling systems, andground-penetrating radar for geological observations.

In some embodiments, the SS technology may be applied to the radarsystems to mitigate “noise”, “interference”, and/or “clutter” at anypoint within the process and thereby enhancing the quality of the finaldata delivered to the end use application. The SS technology may beintroduced independent of any other correction algorithms and systems orin conjunction with one or more of such systems, such as: pulse-doppler,moving target indication, automatic gain control (“AGC”), 3D mappingimaging applications, and/or horizontal, vertical, linear and circularpolarization. Reflected signals decline rapidly as distance increases,so noise introduces a radar range limitation, and the lower the power ofthe desired signal, the more difficult it is to discern it from thenoise. Radar systems must be configured to overcome unwanted signals,that is to say both passive signals and active signals, while focusingon the actual targets of interest. Overcoming unwanted signals maydefine a radar system's signal-to-noise ratio (“SNR”), comparing thelevel of a desired target signal to the level of background noise orinterference.

In an exemplary embodiment, introduction of SS technology may increase aradar system's SNR that may result in delivering improvements inisolating actual targets from the surrounding noise signals,interference and clutter. In an example, noise and interference may becaused by any of the factors including, internal source of randomvariations in the signal, which may be generated by all electroniccomponents; random variations superimposed on the desired echo signalreceived in the radar receiver; and/or external sources, such as thermalradiation of the background surrounding the target of interest. Inaddition, clutter may be caused due to radio frequency echoes returnedfrom targets which are uninteresting to the radar operators. Suchtargets may include natural objects (such as rain, birds); atmosphericturbulence and other atmospheric effects (such as ionospherereflections); man-made objects (such as buildings); and/or even radarcountermeasures such as chaff. Some clutter may also be caused by a longradar waveguide between the radar transceiver and the antenna. The SSmethods and techniques described herein may serve to effectivelymitigate interference from the above and other interfering signals. TheSS technology may be applied to all forms of radar signals, equipmentand imaging software and hardware, regardless of frequency bands, scantypes, display processors and systems utilized, and/or end uses andlinks. The technology may also be applied to other systems that make useof other parts of the electromagnetic spectrum. One example of such asystem may be “LIDAR”, which uses visible light from lasers rather thanradio waves. In addition, the technology may be applied to otherradiofrequency-based (RF) systems, such as a scalable multifunction RFsystem which enables RF functionality (e.g. radar, communications, andelectronic warfare) to be extended, identified, separated, concealed orotherwise manipulated in the performance of its functions.

In accordance with an exemplary and non-limiting embodiment a sourceseparated signal generated using any process or combination of thepreviously described techniques herein may generate outputs presentedas: (i) an audio file; and/or (ii) audio signal components; and/or (iii)speech feature vectors, all of which alone or in combination can serveas the inputs to a speech recognition engine or biometric voiceidentification system. In some embodiments, the signal separationtechnology described herein may be utilized in speech recognitionsystems which may be used to such as, translate spoken words into text,control automated systems through voice translation, or convert spokenwords into other outputs other than voice through an automated process.Introduction of SS to improve speech and voice recognition may beapplied independently of any other algorithms and systems used toimprove recognition, or in conjunction with one or more of such systems.Additionally, SS may be applied such as to original voice source signalsthat may have been converted to digital signals and reconverted toanalog signals prior to once again being converted to digital to beprocessed for speech recognition, or, to the audio signal once it mayhave been converted to digital format immediately prior to the speechrecognition process.

Speech recognition may be referred to as “automatic speech recognition”(“ASR”), “computer speech recognition”, and/or “speech to text”. Thesesystems may use training, such as in the case of “Speaker Dependent”systems or not use training by a speaker (referred to as “SpeakerIndependent” systems). Voice recognition generally refers to finding theidentity of who is speaking, in contrast to what they are saying.Recognizing the speaker may simplify the task of translating speech inspeaker dependent systems or it may be used to authenticate or verifythe identity of a speaker as part of a security process. In someembodiments, applying SS to speech recognition may include conversion ofanalog voice signal into digital audio and then into recognized speech.In an example, the conversion may be performed through a process thatmay include, transforming the digital audio into a better acousticrepresentation, applying rules so the speech recognizer knows whatphonemes to expect, and determining which phonemes are spoken, andconverting the phonemes into words. The digital audio format may vary interms of number of channels (such as mono vs. stereo), bitrate, and/orother characteristics.

Speech recognition may also include extracting feature vectors fromspeech waveforms. The extraction may be achieved by first transformingthe digital audio into the “frequency domain” using a windowedFast-Fourier Transform (FFT), with a resulting output similar to what aspectrograph produces. In this domain, the frequency components of asound for a given sample rate may be used to generate a graph of theamplitudes of frequency components for that sample. A feature vector maybe computed from a window of speech signals in every short timeinterval, and an utterance may be represented as a sequence of thesefeature vectors.

In some embodiments, an automated speech recognizer engine may consistof a database of thousands of such graphs correlated to different typesof sounds produced by the human voice, and the graph generated at thatsample may be matched against the database, producing a number thatdescribes the sound. The most likely word sequence for the given speechfeature vectors is found using two types of knowledge sources, that isto say, acoustic knowledge and linguistic knowledge. Speech recognitionengines may use a mathematical technique called “Hidden Markov Models”(HMMs) for the acoustic features of speech sound and the stochasticlanguage model may be used to represent linguistic knowledge. In someexamples, interferers such as loud background noise or other ambientenvironmental sounds may often lead to misinterpretation of the source,resulting in the recognizer to determine a different vector than itwould have if the user were in a quiet room with a high-qualitymicrophone. Traditionally, background noise and variability problemshave been addressed using statistical models to figure out which phonemeis spoken; however, with strong interference the results are generallypoor. In an embodiment of applying SS technique to speech recognitionprocess, introduction of SS in the initial steps of the recognitionprocess, whereby the feature vectors are extracted from speech waveformsmay greatly increase the robustness of determining the phonemes andutterances with a much higher confidence than other approaches.Application of SS may greatly mitigate the impact of interferers likeambient noise when extracting the feature vectors from the digital audiosignal. SS processed signals may offer higher accuracy for voicerecognition/identification and may be introduced into any existing voicerecognition or voice security system, using either onboard processing(as with cell phone, tablet and other personal device security features)or linked to a network or cloud for controlled access devices or areas(such as restricted access facilities, buildings, vaults or othersecured locations). For voice/speaker recognition, similar processes maybe used to extract feature vectors of the speaker of interest; however,these vectors may be compared and contrasted to a model/library ofutterances originally created by the speaker, and a similarity score maybe generated. The SS technology may be introduced to voice recognitionto enhance the robustness of the scoring by mitigating interference suchas background noise or competing conversations by delivering improvedfeature vectors through any of the methods including but not limited toapplication to the original voice source signals that have beenconverted to digital signals and reconverted to analog signals prior toonce again being converted to digital to be processed for speechrecognition, or, application to the audio signal once it has beenconverted to digital immediately prior to the voice recognition process.SS methods and systems described herein may be implemented as hardwareor software on any PC, cell phone, tablet, or other system usingvoice/speech recognition, as a stand-alone processing technique or anadd-on to existing software program.

In accordance with exemplary and non-limiting embodiments, arepresentation of signal elements may be developed in a model of asignal. The signal may be decomposed and grouped into tracks and/ortracklets corresponding to individual speakers, and the decomposedsignal transformed into feature vectors adapted for use in a speechrecognition engine. In such embodiments, one might develop and introducea bias toward a specific speaker (e.g. the owner of a phone), so as toautomatically pull out their speech and enhance it over all other soundsin the environment.

In another embodiment, a representation of signal elements which may bereferred to as speech features or speech vectors may be developed in asource signal separation model of a signal. The signal may then bedecomposed into speech feature vectors corresponding to individualspeakers, and the decomposed representation used as an input to a speechrecognition engine or biometric voice identification system.

In accordance with exemplary and non-limiting embodiments, a systemcomprises a sound gathering device, such as a microphone, with a nearbyprocessor for engaging in cooperative/distributed computing of sourcesignal separation. In some embodiments, the algorithm is scalable to beless processing-intensive so it can be used on cellular phones,smartphones, tablets or other mobile devices. In some embodiments, someof the processing may be conducted on the mobile device and then bedistributed or transmitted to a remote processor or server with resultsthen delivered back to the mobile device.

In some embodiments SS techniques may be used for hearing aidapplications. A hearing aid is any medical device that helps amplify andfilter sounds to enable those with hearing impairments/hearing loss tocomprehend sound. Hearing aids consist of microphones (directional oromnidirectional) that may convert sound to an electrical signal, whichmay then be processed by a digital signal processor to enhance targetedsounds and minimize unwanted background noise. The resulting targetedsounds are then amplified and rebroadcast via speakers in the patient'sear canal. Patient controls may be used for volume, noise reduction, anddifferent environmental settings. Microphones, DSPs and controls for thedevice may be located on or within the hearing aid itself or in externalcontrol devices or cell phones.

In some embodiments, the methods for source signal separation describedherein may be embodied in any design hearing aid device for the purposesof, but not limited to, amplifying targeted sounds, focusing on a singleperson speaking or sound source, focusing on limited region, such as aconversation at a table in a crowded restaurant while turningoff/minimizing other sounds in the restaurant, and/or minimizing oreliminating background or other ambient noises that the user may choosenot to hear and/or interfere with his/her comprehension of a desiredconversation or sound source. These SS methods may be employed acrossany hearing assistance device including but not limited tobehind-the-ear aids, in-the-canal hearing aids, open canal aids, closedcanal aids, air conduction hearing aids, bone conduction/bone anchoredaids, eyeglass based aids, external device-linked aids, cell phone basedaids, PDA based aids, iPad/Tablet based aids, PC based aids and cochlearimplants. The SS techniques may also be applied in hearing assistancedevices includes both FDA-Regulated hearing aids and over-the counternon-prescription sound amplification devices.

In some embodiments, the SS methods described herein may also be linkedto cell phone, television, radio, PC, Cloud, tablet and otherhearing-assistance linked devices. One exemplary embodiment may belinkage to a television to enable the user to comprehend the broadcastwhile minimizing or turning off other background or ambient noises thatmay impair a user's ability to comprehend the broadcast. Likewise asimilar embodiment of this application may include the amplification ofa cell phone transmission processed to minimize or eliminate ambient orbackground noises both at the site where the user is receiving the callas well as the unwanted background noises transmitted by the caller onthe other end of the line.

In some embodiments, the SS methods described herein may be intended towork with any microphone (stereo or mono, directional oromni-directional) or microphone array located on or incorporated intoany hearing assistance device, or located off the hearing assistanceprocessing device and transmitted to that device via wireless, infra-red(IR), Bluetooth, wired or other transmission methods. An exemplaryembodiment may be a cell phone or tablet linked hearing aid where soundmay be recorded on these devices and them transmitted to the ear forbroadcast. Likewise, microphones for recording targeted sound sourcesmay be located on the users eyeglasses, embedded into clothing orjewelry, worn around the user's neck, embedded in buttons, hats or otherclothing or fashion accessories. Microphone designs, including but notlimited to the above examples, may transmit targeted sounds to aprocessing device, where the SS methods and system described herein maybe configured to process those sounds. The algorithm processing may takeplace on an independent DSP or in the device's CPU through embeddedfirmware. The deployment of these processing platforms may be on thedevice itself, an external control unit, a tablet, PC, PDA, cell phoneor transmission through a cloud or transmission back to a central serverover a cellular or wireless network. Signals recorded on bilateralhearing aids or array microphone systems may be transmitted acrossdevices or to an external processing unit, including but not limited tothose described above, for real time or near-real time processing.

In some embodiments, signals processed with the SS techniques describedherein may then be re-synthesized into an output signal to be playedback through a speaker in or near the users' ear, or through an neuralor bone stimulation device for direct sensoneural processing. Speakerbased devices for rebroadcast include open canal and closed canalsystems, headphones, telephonic devices, cell phones, Bluetooth andother speaker based devices. Re-synthesized signals may be captured onthe same device (such as a behind the ear hearing aid) or transmitted tothe output speaker devices from an external processing unit (such as atablet, cell phone, PC or other portable processor) and may be a singlereprocessed input or the combination of many simultaneously recorded andmixed inputs from multiple recording devices. Hearing assistancetechnologies making use of SS processing may feature clinical programmedparameters or user controlled parameters to adjust device processing toa specific environment. An exemplary embodiment of clinician parameterswould be distance based SS and background noise reduction setting thatmay be programmed at the time of the initial fitting or subsequentlyadjusted via telephonic or PC/web interface reprogramming. An exemplaryembodiment of user based controls may include onboard device dials,external control units, or PC/cellphone/Tablet based applications thatmay allow the user to control the mix of targeted speech to backgroundnoise, the level of targeted speech amplification, the use of real-timeor near-real-time transmission, distance and vector based controls togovern the area or direction in when they would like to gather targetedsound sources, the ability to tap into TV, cell phones, radios, voicecontrol systems or other PC based devices for direct interface. Usersmay also have the ability to set the device for various modes, such asrestaurants or close conversations, or control the lead-in time forplayback such that they may determine tradeoffs between delayed lead-insfor targeted speech vis-a-vis intelligibility or naturalness ofrebroadcast sounds.

In accordance with exemplary and non-limiting embodiments, a systemcomprises a sound gather device, such as a microphone, or a soundtransmitting device for communication (e.g., using Bluetooth or othertransmission protocol), with a nearby processor for engaging incooperative/distributed computing of source signal separation. In someembodiments, the algorithm is scalable to be less processing-intensiveso it can be used on hearing aids. In some embodiments, some processingmay be distributed to remote server by the processor with resultsforwarded to the hearing aid.

In one variation, a cell phone can send data to a server that canperform more processing. In some instances, as when a hearing aid reallyneeds more processing power and it can't transmit to a remote server, itmay transmit to a nearby device such as a phone in your pocket. Thephone may act like a local super booster or external processing system.In such an instance, the hearing aid could transition to a defined modeand use extra computing power to offload processing to the cell phoneand achieve improved capabilities. In one example, controls may beplaced on an actual cell phone or computing tablet such that, forexample, a person sitting in a restaurant can put the cell phone down onthe table and can tap a screen or move a slider control to tailorprocessing and source signal separation in the directions of the peoplesitting at the table. In response, an algorithm operates to help enhancetable-mates conversation. FIG. 15 illustrates an exemplary andnon-limiting representation of such a computer generated interface fortablet or cell phone control.

In another embodiment, ambient noise or unwanted background noise may beremoved from an input source signal to produce a deconstructed sourcesignal which then may be re-combined with the ambient or backgroundnoise at a lower noise level, and outputting the combined signal. Insome embodiments, the user may dynamically or statically alter the noiselevel of the ambient noise re-introduced.

In some embodiments, the SS techniques described herein may be used intelephony applications. For mobile phone calls on cellular networks, theaudio is captured through an embedded microphone and is subsequentlyconverted from an analog to a digital signal (typically referred to asan “A to D” conversion). The resulting digital signal is thentransmitted through the cellular network in a compressed ornon-compressed form to an end terminus whereby it is delivered as audiooutput. Anywhere along the transmission process or at the endpoint ofdelivery, the digital signal is converted back to an analog signal.Typically, audio captured by a phone (such as a cellular phone, aspeakerphone, a VoIP phone and the like) for sending may contain ambientnoise or other interferences which will not inhibit the conversion nortransmission of the audio file, but may impact the general quality ofthe output file to the intended receiver. For example, the microphone ina mobile phone may pick up the voice of the speaker, but may also becapturing the noise of other conversations occurring near the caller ofinterest, which may be converted and transmitted to the receiver of thecall. When the audio is converted and delivered to the receiver, thelistener may find it difficult to understand the speaker with theinterfering noise also delivered. Generally certain algorithms such asnoise and echo cancellation are applied at the point of capture (such asin the mobile phone), where the signal may be converted fortransmission, however, the applied algorithms traditionally onlymitigate some of the noise/interfering effects and the receiving partymay still receive interfering environmental noises which may impede theperceptibility of the sender.

The methods for source signal separation described herein may beintroduced into any telephony application for the purposes of, but notlimited to, amplifying targeted sounds and/or focusing on the cell phoneor telephone user or the person of interest speaking on a conferencecall while minimizing or eliminating background or other ambient noisesthat a receiving party would prefer not to hear and/or have transmitted,as such unwanted transmissions would interfere with the user'scomprehension of the calling party, speaker of interest and/orconversation.

These SS methods may be introduced and applied during any point of thesource signal capture, conversion, transmission and/or delivery/outputto the receiver in a telephony application. The SS methods may beintegrated to be always applied during a call, or may be introduced witha control mechanism that may enable the sender or receiver to requestthe introduction of the SS methods to provide mitigation of interferersduring a call. SS systems and methods may be incorporated as firmware,embedded software, a stand-alone software application or platform, or anadditional software function or feature, which may be implemented fromthe point of collection, transmission or delivery (such as a cell phoneor network) to be used alone or in conjunction with other algorithms fornoise reduction, call clarity and/or other performance benefits.

In some embodiments, the SS applications may be used in car voicecontrol systems that may face challenges in processing elements of atargeted audio command mixed with any of the following or similarinterfering sound sources: road noise, external environmental noise,radio noise, HVAC noise, unintended cabin noise and accompanyingpassenger noises. The SS methods described herein may be used inconjunction with in-car voice response systems to extract and amplifytargeted commands from unwanted or interfering background noise foraccurate voice response system processing, automotive controls andvehicle security. The SS methods described herein may interact with avoice command system through the use of speech or extracted speechfeatures that may be processed by the voice response system. Theprocessing system may be contained on-board in a car-based-PC ortransmitted to a central processing server outside of the vehicle. Anexemplary embodiment of the voice response commands controlled by such asystem may include but are not limited to in-car navigation, auto systemcontrols such as HVAC, windows, radio, seat function, wipers, automaticdoor locks and controls, sunroof controls and third party integrateddevice controls such as cell phone integration and iPod, tablet, mp3,audio and entertainment device controls. The SS system may also belinked to cellphones; Bluetooth and other headset systems to processsuch as both send and receive signals that may be passing through thevehicles central audio processing system. An additional deployment ofthe SS methods may be onboard voice biometrics for vehicle controls andsecurity. Speech features captured by the SS systems and methodsdescribed herein may enable the extraction of precise speech featuresunique to each individual user. Representative deployments of thiscontrol feature may include but are not limited to driver/user assignedvehicle locks and alarm controls, driver engine start and turn-offcontrols (initiated onboard or through an external control device suchas a cell phone), driver and/or specific user controls of navigationsystems and non-essential vehicle control systems.

The SS systems described herein may be enabled by a single microphone(stereo or mono, directional or omnidirectional) or an array ofmicrophones built into the cabin or through linkage to an externalsystems such as a Bluetooth headset or other hands free cellphonecontrol device. The system may be deployed and programmed by the usersuch that the voice control system may only accept prompts for thedriver's seat, both the driver and passenger seats, or an individualwith a designated biometric signature. Separate controls may also beadded such that individuals in the rear seats may control rear HVACsystems or rear entertainment systems. In some embodiments, drive oradditional party voice biometric controls may be programmed through useof the system or through a download user voice biometric profile fromanother device using the SS methods described herein.

In some embodiments, the SS methods described herein may be deployed ina series of medical imaging applications that make use of static imagingor time-series imaging signal analysis including but not limited to thefollowing: Ultrasound, MRI, CT Scans, PET Scans, X-Rays, SPECT, GammaCamera Imaging, Nuclear Imaging, Photoacoustic Imaging, BreastThermography, and Optical Coherence Tomography. The application of theSS methods described herein may enable improved resolution of targetedimages and the reduction of noise generated by the imaging equipment inthe above mentioned and other medical imaging systems. An exemplaryembodiment of the SS methods and systems described herein may includeapplications in medical ultrasound systems to enhanced resolution andreduce the noise generated by overlapping elements in the ultrasoundprobe. SS algorithms may be incorporated into freestanding ultrasoundsystems, pc-based systems, tablet systems, smart phone apps, PDAs, andhandheld systems. The SS algorithms may be incorporated as firmware thatmay run off the devices internal CPUs, software, or apps loaded on tothe devices, or as DSPs or other chips incorporated into the control boxor onto the ultrasound probe itself. The SS methods and systems forimproved ultrasound may be incorporated pre- or post-summation of thedata collected by the individual elements in the probe. The SS methodsand systems described herein may be used pre and/or post beam formationso as to be compatible with adjustments in beam angles and signalintensity to compensate for differences in targeted anatomy.

In some embodiments, the SS methods and systems described herein may beused with any form of ultrasound (such as sonography or echosonography)imaging software or add-on imaging analysis programs including but notlimited to 2D ultrasound, 3D ultrasound, 4D ultrasound, tissue doppler,flow doppler tissue strain analysis, elasticity analysis and otherapplications. The SS software may be applied across all clinicalpractices including both diagnostic and surgical applications.Embodiments of SS enhanced ultrasound image may include ultrasoundassisted biopsies, ultrasound assisted catheter placement, echocardiology, cardiology and cardiac surgery applications, orthopedic andorthopedic surgical applications, sonography and other obstetrics andgynecology applications, including both imaging and surgical, urologicalapplications, gastrointestinal applications, soft tissue applications,head, neck and cranial applications. The core ultrasound applicationsdescribed herein may also be used with both ultrasound hardware andimaging software programs for veterinary and industrial applicationsincluding but not limited to ultrasonic analysis of composite materials,structures, and geological surveys.

In some embodiments, the SS techniques may be used for applicationsrelated to Sound Navigation And Ranging (Sonar), as well as for hydroacoustics applications. Sonar uses sound propagation to navigate,communicate with and/or detect objects on or under the surface of thewater. There may be two types of sonar based applications that mayinclude, applications based on a passive sonar technology that may“listen” for sounds generated by target objects; and applications basedon an active sonar technology that may emit pulses of sounds and listenfor echoes. Sonar may be used as a means of acoustic location and ofmeasurement of the echo characteristics of “targets” in the water, andmay be used in applications including, but not limited to, submarinenavigation, guidance for torpedoes and mines, underwater survey andmapping, echo sounding, pipeline inspection, wave measurement, anddetermining the course, range, trajectory and speed of a target ofinterest (such as using the Target Motion Analysis).

In some embodiments, the SS methods and systems described herein may beused to enhance the signal quality with any form of active sonar whichmay use a sound transmitter and a receiver, which may be operated inmonostatic, bistatic or multistatic configurations and the acousticfrequencies may vary from very low (infrasonic) to extremely high(ultrasonic). The sonar may utilize a pulse of sound generally createdelectronically using a signal generator, power amplifier andelectro-acoustic transducer/array at constant frequency or a “chirp” ofchanging frequency (enabling pulse compression upon reception). The SSmay also be incorporated in conjunction with a beam former that may beused to concentrate the acoustic power into a beam, which may be sweptto cover the required search angles. Occasionally, the acoustic pulsemay be created by other means, such as by chemically using explosives,or by using air guns or by using plasma sound sources.

In some embodiments, the SS methods and systems described herein may beused to enhance the signal quality with any form of passive sonar, whichmay typically “listen” without transmitting any pulses and has a widevariety of techniques for identifying the source of a detected sound,generally by comparing the detected sound against large sonic databases.Through use of passive sonar, if the target radiated noise level is highenough it allows the target to be identified. However, in some examples,operation may be affected by variations in sound speed determined by thewater's bulk modulus, mass density, temperature, dissolved impurities(usually salinity), and even water pressure.

In one or more embodiments described herein, the SS methods describedherein may be applied to all forms of active and passive sonar systemsto address sound variations as well as mitigate noise, interference,and/or scatter at any point within the process of analysis once sound orecho has been received, and thereby enhancing the quality of the finaldata delivered to the end use application. It may be introduced in thesoftware or hardware components of the receiving, transmission ordisplay systems independent of any other correction algorithms andsystems or in conjunction with one or more of such systems, such as beamforming and narrow beam transmissions. In some examples, sources ofnoise that interfere with the desired target echo or signature may rangefrom waves and shipping to turbulence and marine life. Additionally, themotion of the receiver through the water may also cause speed-dependentlow frequency noise. When active sonar is used, scattering may occurfrom small objects in the sea as well as from the bottom and surface. Inaddition to active and passive sonar, the SS technology may be appliedto deliver benefit to other sonar-based systems including, but notlimited to, synthetic aperture sonar and parametric and non-linearsonar. The SS methods and systems described herein may also beintroduced to hydro acoustic systems, including underwater acousticcommunication that may be used to send and receive messages below water.There may be several ways of employing such communication but the mostcommon may include using hydrophones. Underwater communication may bedifficult due to numerous factors, which can be addressed by SS,including but not limited to: multi-path propagation; time variations ofthe channel; small available bandwidth; and strong signal attenuation.

In some embodiments, the SS systems and methods described herein may beused in microphone dependent systems. Much like cell phones and othertelephony systems, headsets, speakerphones and general microphone basedsystems (used either alone or in conjunction with cellular or othertelephony networks) may have the unintended effects of receiving,processing and transmitting the device user as well as unintendedbackground noise and ambient noise present at the time oftransmission/recording. Current systems may not be capable of isolatingthe targeted users from other ambient or interfering noises that mayoverpower the speaker and may make it difficult for the receiver/user tocomprehend the intended transmission/recording. Representative examplesof this problem may include: the transmission of airplane noise throughflight control systems, the broadcast of PA announcements at the airportthrough a cell phone headset, room noise broadcast through a conferencecall speaker system, auto and outdoor noises broadcast through a“drive-thru” ordering system, or even crowd noise broadcast over acoach's headset.

The SS systems and methods described herein may be incorporated intosuch microphone dependent devices for the purpose of improving thequality/intelligibility of the user relative to unwanted/unintendedambient/background noises captured by the microphone in thetransmitting/recording device. The SS methodology may be optimized foreach device so that it may only transmit sound sources emanating from aspecific speaker, or defined limited area/radius, such as the proximaldevice user and turning off far field noises. This may be achieved byusing SS methods and systems to extract and selectively transmit/recordsounds from the targeted speaker and not the unintended backgroundnoises.

An exemplary embodiment of such a system may include the addition of theSS systems and methods described herein to a Bluetooth headset. The SStechnology may be added to the headset as a dedicated DSP or firmwareadded to an existing processor. It may be capable of processing thesignals captured by the devices' microphone (directional oromni-directional), extracting the targeted sound source from theunintended noise, before the resulting signal may be transmitted orrecorded. This may assure that the recording device or transmittingsystems may only record the extracted sound source, hence increasing thequality and intelligibility of that sound source. This new step in theprocessing chain may be used as a stand-alone feature or may be used incombination with other audio processing and enhancement algorithms.Another exemplary embodiment of the SS systems and methods describedherein may be used in microphone-based recordings. Targeted soundsources may be extracted and recorded on one channel, while backgroundnoises may be recorded on a separate channel. Each channel may then beremixed for optimal/desired sound effects and quality.

In accordance with exemplary and non-limiting embodiments, a systemcomprises a sound gathering device, such as a microphone, or a soundtransmitting device for communication (e.g., using Bluetooth or anothercommunications protocol), with a nearby processor for engaging incooperative/distributed computing of source signal separation. In someembodiments, some processing may be distributed to remote server by theprocessor with results returned and transmitted through thecommunication system.

In another embodiment, ambient noise or background noise distinct fromthe targeted input signal may be removed from an input source signal toproduce a deconstructed source signal which may then be re-combined withthe ambient or background noise at a lower or reduced presentation leveland outputting the combined signal. In some embodiments, the user maydynamically or statically alter the presentation level of thereintroduced ambient noise.

In some embodiments, the SS systems and methods described herein may beused in voice controlled television and other interactive device basedapplications. The growth of voice recognition and voice driven commandsystems for TV, video games, entertainment systems and other interactivedevices has been limited by the challenges of interfering noises,unintended speakers interrupting commands, and background noiseimpacting command recognition and response. The SS methods describedherein may be embedded in any such entertainment device for the purposeof assuring accurate voice recognition and response. Additionally, suchdevices may be linked or utilize a network-dependent solution for speechand voice recognition similar to those described {in the sectiondetailed earlier} to which SS methods described herein may be applied.An exemplary embodiment of the SS systems and methods described hereinmay include be the use of SS in voice response/voice controls fortelevision function. SS may enable the system to focus on a specificspeaker (s) that may be preprogrammed in the system or an unknownspeaker talking into a remote control or other similar device. Thespeakers' voice commands may be configured to control all devicefeatures and those of related devices including but not limited to cableTV boxes, DVR systems, satellite systems, DVD players, integrated soundsystems, PCs, video game systems/consoles, internet connectivity, cloudconnectivity, video conference systems, VOIP/internet phone systems, andother similar devices. In some examples, the TV voice response controlsmay be driven by any microphone or speaker/microphone combinationsystems including but not limited to television embeddedmicrophone/speakers, dedicated remote control microphone/speakers,external microphone/speaker systems, cell phones, tablets, PCs, videogame systems and headsets. In such examples, the control features mayuse directional/omni-directional microphones and or may make use of IR,bluetooth, wifi, RF or wired linkages to the system. Such a system maypermit two-way interaction, both accepting and responding to voicedriven queries, and it may also serve as the interface for videoconferencing, web conferencing, VOIP, and web based conference calls.The SS methods and systems for Voice Controlled TV described herein mayor may not re-synthesize the received speech. In noisy environments,received speech may be processed as speech features or speech vectorsbased on the SS mathematical models described herein for purposes ofdriving a speech recognition engine or voice response system. Withre-synthesized speech, varying levels of background noise may bereincorporated. The system may be trained to respond to a targeted voiceor voices. In some embodiments, speaker recognition training may begenerated through device use or the citation of speech at the time ofdevice initialization.

In some embodiments, the SS methods and systems described herein may beused in electrical power supply monitoring related applications. Theelectrical power supply emits a continuous low-level noise, which e.g.,averages roughly 50 Hz in some applications. Fluctuations in powerdemand may cause slight variations in this noise level. For example,increased electrical demand may lower the noise level, while reduceddemand level may have the opposite effect. Fluctuations in power demandmay give the power grid the capability of providing a unique time/datesignature that may be correlated with any recording. The SS systems andmethods described herein may be used to monitor the electric grid tocreate a highly accurate time series signature of the system. Thissignature may be derived from any recording device (audio or video) orsource signal type (analog or digital). The low level audio signal maybe consistent across the system and the signal analysis may take placeat generation station, specific machine or any other location. The SSsystems and methods described herein may be configured to extract thesignal impact of electrical supply from any live feed or recording toprovide a highly accurate time series signature of the electrical grid.This signature may be monitored real-time, near real-time orsubsequently analyzed. An exemplary embodiment of this system may use SSto predict impending brown-outs, power spikes, power failures ordisruptions in power supply. This may occur at a grid-wide level, at anindividual site, or on an individual device by analyzing changes in thelow-level noise vs. historic standards/predictors. A recording devicemay record the ambient noise at any of the above locations, machines ordevices and then SS methods would separate the targeted electrical noisefrom other noises. The SS methods and systems may be configured togenerate a reading of the power noise and send a warning of an impendingevent if the noise level poses any concerns. The warning may appear onthe device itself or be sent through a network, wireless or through thecloud to any monitoring device, PC, tablet, cell phone or any otherdevice.

Another exemplary embodiment of this system may be related to forensicaudio analysis. This embodiment may include identification andvalidation of the date and time during which a recording was created.The SS methods and systems described herein may be used to extract theelectrical system noise from a recording and generate a highly accuratemathematical representation of that signal. That signal may becorrelated to known recordings from the electrical grid to determine theexact time and date at which the recording was created. Suchauthentication and/or validation may be necessary for verifyingrecordings to be admitted into evidence and to assure that suchrecording have not been adulterated. The analysis may be conducted onany type of recording (such as audio or video, digital or analog), fileformat, or duration of recording.

Fit User Interface allows a user to view and interact with the tracking,grouping, and peak selection for resynthesis stages of processing. Userinterface may be used “offline” to view and modify stored data, or“online” to command the processing components and interact with the datain real time. It may be used to analyze data, and modify componentparameters. It may detect optimal component parameters from userinteraction. For example, given a user's selection of data forresynthesis, the Fit User Interface may calculate processing parametersfor detecting similar data.

In accordance with an exemplary and non-limiting embodiment, a userinterface is provided for viewing a signal as: tracks; a plurality ofpotentially coherent tracklets and/or coherent groups for editing thevisual representation to at least one of add, remove or group signaldata with the tracks, tracklets and/or coherent groups.

In another embodiment, the user interface may be utilized to view asignal as: tracks; a plurality of potentially coherent tracklets; and/orcoherent groups wherein a user can click on a track, tracklet; and/orcoherent group and to be presented the data associated with that track,tracklet and/or coherent group. In another embodiment, the userinterface may be utilized for viewing a signal as a track; plurality ofpotentially coherent tracklets; and/or coherent groups wherein a usercan search and find a track and/or tracklet within the interface basedon input comprising characteristic data about that track, tracklet,and/or group. In another embodiment, a user may change the scoringfunction on the fly to modify what data is associated into tracks,groups, and/or tracklets.

With reference to FIG. 16, there is illustrated an exemplary embodimentof a track editor as may be practiced in accordance with the embodimentsand description above. As illustrated, the track editor displays aplurality of tracklets composed of oscillator peaks. In variousexemplary and non-limiting embodiments, oscillator peaks may be coloredaccording to track-id. In yet other embodiments, oscillator peaks may becolored according to coherent group-id. In other embodiments, oscillatorpeaks may be colored or set transparent according to whether or not theyare selected for resynthesis. In other embodiments, oscillator peaks maybe colored according to any other oscillator peak parameter. In otherembodiments, oscillator peaks may be scaled according to amplitude,amplitude with respect to background power, or with equal size.

With reference to FIG. 17, there is illustrated an exemplary andnon-limiting embodiment of a track editor GUI. In accordance withexemplary and non-limiting embodiments, a user may select data displayedin the track editor GUI in order to perform an action on the selecteddata. In one embodiment, data may be selected by area such as viadrawing with a box or a lasso. In other embodiments, a user may selectdata by tracklet such as by clicking on any peak in a tracklet. In otherembodiments, a user may select data by coherent group such as byclicking on any peak in a coherent group. In yet another embodiment, auser may select data by oscillator peak such as by clicking on any peak.

Once selected, a user may select an action to be performed on the data.For example, a user may plot the data in another view wherein there isvisually rendered oscillator peak statistics, direction of arrival,time-domain audio, spectrogram data and the like. In some embodiments, auser may Instruct the system whether or not to include select peaks forre-synthesis such as via a “Turn on/Turn off” option.

With reference to FIG. 18, there is illustrated an exemplary embodimentof a data visualizer for displaying user selected data as describedabove.

The SS methods and systems in accordance with various embodiments may beimplemented in software, hardware, firmware, or any combination thereof.The processes may preferably be implemented in one or more computerprograms executing on a variety of computer-equipped devices (such aspersonal computers, mobile phones, imaging devices, hearing aids,interactive voice response systems, conference call systems, audiorecording devices, in-vehicle voice activation systems, dictationsystems, and communications systems). Such devices may include, amongother things, a computer processor (such as general and special purposemicroprocessors), and a storage medium readable by the processor andinput and output devices. Each computer program may be a set ofinstructions (program code) in a code module resident in the randomaccess memory of the device. Until required by the computer processor,the set of instructions may in some cases be stored in another computermemory (such as in semiconductor memory devices, hard disk drives, orremovable memory devices such as optical disks, external hard drives,memory cards, or flash drives) or stored on another computing device anddownloaded via the Internet or other network.

Having thus described several illustrative embodiments, it may beappreciated that various alterations, modifications, and improvementswill readily occur to those skilled in the art. Such alterations,modifications, and improvements may be intended to form a part of thisdisclosure, and may be intended to be within the spirit and scope ofthis disclosure. While some examples presented herein involve specificcombinations of functions or structural elements, it should beunderstood that those functions and elements may be combined in otherways according to the present disclosure to accomplish the same ordifferent objectives. In particular, acts, elements, and featuresdiscussed in connection with one embodiment are not intended to beexcluded from similar or other roles in other embodiments. Additionally,elements and components described herein may be further divided intoadditional components or joined together to form fewer components forperforming the same functions.

While only a few embodiments have been shown and described, it will beobvious to those skilled in the art that many changes and modificationsmay be made thereunto without departing from the spirit and scope asdescribed in the following claims. All patent applications and patents,both foreign and domestic, and all other publications referenced hereinare incorporated herein in their entireties to the full extent permittedby law.

The methods and systems described herein may be deployed in part or inwhole through a machine that executes computer software, program codes,and/or instructions on a processor. Various embodiments described hereinmay be implemented as a method on the machine, as a system or apparatusas part of or in relation to the machine, or as a computer programproduct embodied in a computer readable medium executing on one or moreof the machines. In embodiments, the processor may be part of a server,cloud server, client, network infrastructure, mobile computing platform,stationary computing platform, or other computing platform. A processormay be any kind of computational or processing device capable ofexecuting program instructions, codes, binary instructions and the like.The processor may be or may include a signal processor, digitalprocessor, embedded processor, microprocessor or any variant such as aco-processor (math co-processor, graphic co-processor, communicationco-processor and the like) and the like that may directly or indirectlyfacilitate execution of program code or program instructions storedthereon. In addition, the processor may enable execution of multipleprograms, threads, and codes. The threads may be executed simultaneouslyto enhance the performance of the processor and to facilitatesimultaneous operations of the application. By way of implementation,methods, program codes, program instructions and the like describedherein may be implemented in one or more thread. The thread may spawnother threads that may have assigned priorities associated with them;the processor may execute these threads based on priority or any otherorder based on instructions provided in the program code. The processor,or any machine utilizing one, may include memory that stores methods,codes, instructions and programs as described herein and elsewhere. Theprocessor may access a storage medium through an interface that maystore methods, codes, and instructions as described herein andelsewhere. The storage medium associated with the processor for storingmethods, programs, codes, program instructions or other type ofinstructions capable of being executed by the computing or processingdevice may include but may not be limited to one or more of a CD-ROM,DVD, memory, hard disk, flash drive, RAM, ROM, cache and the like.

A processor may include one or more cores that may enhance speed andperformance of a multiprocessor. In embodiments, the process may be adual core processor, quad core processors, other chip-levelmultiprocessor and the like that combine two or more independent cores(called a die).

The methods and systems described herein may be deployed in part or inwhole through a machine that executes computer software on a server,client, firewall, gateway, hub, router, or other such computer and/ornetworking hardware. The software program may be associated with aserver that may include a file server, print server, domain server,internet server, intranet server, cloud server, and other variants suchas secondary server, host server, distributed server and the like. Theserver may include one or more of memories, processors, computerreadable media, storage media, ports (physical and virtual),communication devices, and interfaces capable of accessing otherservers, clients, machines, and devices through a wired or a wirelessmedium, and the like. The methods, programs, or codes as describedherein and elsewhere may be executed by the server. In addition, otherdevices required for execution of methods as described in thisapplication may be considered as a part of the infrastructure associatedwith the server.

The server may provide an interface to other devices including, withoutlimitation, clients, other servers, printers, database servers, printservers, file servers, communication servers, distributed servers,social networks, and the like. Additionally, this coupling and/orconnection may facilitate remote execution of program across thenetwork. The networking of some or all of these devices may facilitateparallel processing of a program or method at one or more locationwithout deviating from the scope of the disclosure. In addition, any ofthe devices attached to the server through an interface may include atleast one storage medium capable of storing methods, programs, codeand/or instructions. A central repository may provide programinstructions to be executed on different devices. In thisimplementation, the remote repository may act as a storage medium forprogram code, instructions, and programs.

The software program may be associated with a client that may include afile client, print client, domain client, internet client, intranetclient and other variants such as secondary client, host client,distributed client and the like. The client may include one or more ofmemories, processors, computer readable media, storage media, ports(physical and virtual), communication devices, and interfaces capable ofaccessing other clients, servers, machines, and devices through a wiredor a wireless medium, and the like. The methods, programs, or codes asdescribed herein and elsewhere may be executed by the client. Inaddition, other devices required for execution of methods as describedin this application may be considered as a part of the infrastructureassociated with the client.

The client may provide an interface to other devices including, withoutlimitation, servers, other clients, printers, database servers, printservers, file servers, communication servers, distributed servers andthe like. Additionally, this coupling and/or connection may facilitateremote execution of the program across the network. The networking ofsome or all of these devices may facilitate parallel processing of aprogram or method at one or more location without deviating from thescope of the disclosure. In addition, any of the devices attached to theclient through an interface may include at least one storage mediumcapable of storing methods, programs, applications, code and/orinstructions. A central repository may provide program instructions tobe executed on different devices. In this implementation, the remoterepository may act as a storage medium for program code, instructions,and programs.

The methods and systems described herein may be deployed in part or inwhole through network infrastructures. The network infrastructure mayinclude elements such as computing devices, servers, routers, hubs,firewalls, clients, personal computers, communication devices, routingdevices and other active and passive devices, modules and/or componentsas known in the art. The computing and/or non-computing device(s)associated with the network infrastructure may include, apart from othercomponents, a storage medium such as flash memory, buffer, stack, RAM,ROM and the like. The processes, methods, program codes, instructionsdescribed herein and elsewhere may be executed by one or more of thenetwork infrastructural elements. The methods and systems describedherein may be adapted for use with any kind of private, community, orhybrid cloud computing network or cloud computing environment, includingthose which involve features of software as a service (SAAS), platformas a service (PaaS), and/or infrastructure as a service (IaaS).

The methods, program codes, and instructions described herein andelsewhere may be implemented on a cellular network having multiplecells. The cellular network may either be frequency division multipleaccess (FDMA) network or code division multiple access (CDMA) network.The cellular network may include mobile devices, cell sites, basestations, repeaters, antennas, towers, and the like. The cell networkmay be a GSM, GPRS, 3G, EVDO, mesh, or other networks types.

The methods, program codes, and instructions described herein andelsewhere may be implemented on or through mobile devices. The mobiledevices may include navigation devices, cell phones, mobile phones,mobile personal digital assistants, laptops, palmtops, netbooks, pagers,electronic books readers, music players and the like. These devices mayinclude, apart from other components, a storage medium such as a flashmemory, buffer, RAM, ROM and one or more computing devices. Thecomputing devices associated with mobile devices may be enabled toexecute program codes, methods, and instructions stored thereon.Alternatively, the mobile devices may be configured to executeinstructions in collaboration with other devices. The mobile devices maycommunicate with base stations interfaced with servers and configured toexecute program codes. The mobile devices may communicate on apeer-to-peer network, mesh network, or other communications network. Theprogram code may be stored on the storage medium associated with theserver and executed by a computing device embedded within the server.The base station may include a computing device and a storage medium.The storage device may store program codes and instructions executed bythe computing devices associated with the base station.

The computer software, program codes, and/or instructions may be storedand/or accessed on machine readable media that may include: computercomponents, devices, and recording media that retain digital data usedfor computing for some interval of time; semiconductor storage known asrandom access memory (RAM); mass storage typically for more permanentstorage, such as optical discs, forms of magnetic storage like harddisks, tapes, drums, cards and other types; processor registers, cachememory, volatile memory, non-volatile memory; optical storage such asCD, DVD; removable media such as flash memory (e.g. USB sticks or keys),floppy disks, magnetic tape, paper tape, punch cards, standalone RAMdisks, Zip drives, removable mass storage, off-line, and the like; othercomputer memory such as dynamic memory, static memory, read/writestorage, mutable storage, read only, random access, sequential access,location addressable, file addressable, content addressable, networkattached storage, storage area network, bar codes, magnetic ink, and thelike.

The methods and systems described herein may transform physical and/oror intangible items from one state to another. The methods and systemsdescribed herein may also transform data representing physical and/orintangible items from one state to another.

The elements described and depicted herein, including in flow charts andblock diagrams throughout the figures, imply logical boundaries betweenthe elements. However, according to software or hardware engineeringpractices, the depicted elements and the functions thereof may beimplemented on machines through computer executable media having aprocessor capable of executing program instructions stored thereon as amonolithic software structure, as standalone software modules, or asmodules that employ external routines, code, services, and so forth, orany combination of these, and all such implementations may be within thescope of the present disclosure. Examples of such machines may include,but may not be limited to, personal digital assistants, laptops,personal computers, mobile phones, other handheld computing devices,medical equipment, wired or wireless communication devices, transducers,chips, calculators, satellites, tablet PCs, electronic books, gadgets,electronic devices, devices having artificial intelligence, computingdevices, networking equipment, servers, routers and the like.Furthermore, the elements depicted in the flow chart and block diagramsor any other logical component may be implemented on a machine capableof executing program instructions. Thus, while the foregoing drawingsand descriptions set forth functional aspects of the disclosed systems,no particular arrangement of software for implementing these functionalaspects should be inferred from these descriptions unless explicitlystated or otherwise clear from the context. Similarly, it will beappreciated that the various steps identified and described above may bevaried, and that the order of steps may be adapted to particularapplications of the techniques disclosed herein. All such variations andmodifications are intended to fall within the scope of this disclosure.As such, the depiction and/or description of an order for various stepsshould not be understood to require a particular order of execution forthose steps, unless required by a particular application, or explicitlystated or otherwise clear from the context.

The methods and/or processes described above, and steps associatedtherewith, may be realized in hardware, software or any combination ofhardware and software suitable for a particular application. Thehardware may include a general-purpose computer and/or dedicatedcomputing device or specific computing device or particular aspect orcomponent of a specific computing device. The processes may be realizedin one or more microprocessors, microcontrollers, embeddedmicrocontrollers, programmable digital signal processors or otherprogrammable device, along with internal and/or external memory. Theprocesses may also, or instead, be embodied in an application specificintegrated circuit, a programmable gate array, programmable array logic,or any other device or combination of devices that may be configured toprocess electronic signals. It will further be appreciated that one ormore of the processes may be realized as a computer executable codecapable of being executed on a machine-readable medium.

The computer executable code may be created using a structuredprogramming language such as C, an object oriented programming languagesuch as C++, or any other high-level or low-level programming language(including assembly languages, hardware description languages, anddatabase programming languages and technologies) that may be stored,compiled or interpreted to run on one of the above devices, as well asheterogeneous combinations of processors, processor architectures, orcombinations of different hardware and software, or any other machinecapable of executing program instructions.

Thus, in one aspect, methods described above and combinations thereofmay be embodied in computer executable code that, when executing on oneor more computing devices, performs the steps thereof. In anotheraspect, the methods may be embodied in systems that perform the stepsthereof, and may be distributed across devices in a number of ways, orall of the functionality may be integrated into a dedicated, standalonedevice or other hardware. In another aspect, the means for performingthe steps associated with the processes described above may include anyof the hardware and/or software described above. All such permutationsand combinations are intended to fall within the scope of the presentdisclosure.

While the disclosure has been disclosed in connection with the preferredembodiments shown and described in detail, various modifications andimprovements thereon will become readily apparent to those skilled inthe art. Accordingly, the spirit and scope of the present disclosure isnot to be limited by the foregoing examples, but is to be understood inthe broadest sense allowable by law.

The use of the terms “a” and “an” and “the” and similar referents in thecontext of describing the disclosure (especially in the context of thefollowing claims) is to be construed to cover both the singular and theplural, unless otherwise indicated herein or clearly contradicted bycontext. The terms “comprising,” “having,” “including,” and “containing”are to be construed as open-ended terms (i.e., meaning “including, butnot limited to,”) unless otherwise noted. Recitation of ranges of valuesherein are merely intended to serve as a shorthand method of referringindividually to each separate value falling within the range, unlessotherwise indicated herein, and each separate value is incorporated intothe specification as if it were individually recited herein. All methodsdescribed herein may be performed in any suitable order unless otherwiseindicated herein or otherwise clearly contradicted by context. The useof any and all examples, or exemplary language (e.g., “such as”)provided herein, is intended merely to better illuminate the disclosureand does not pose a limitation on the scope of the disclosure unlessotherwise claimed. No language in the specification should be construedas indicating any non-claimed element as essential to the practice ofthe disclosure.

While the foregoing written description enables one of ordinary skill tomake and use what is considered presently to be the best mode thereof,those of ordinary skill will understand and appreciate the existence ofvariations, combinations, and equivalents of the specific embodiment,method, and examples herein. The disclosure should therefore not belimited by the above described embodiment, method, and examples, but byall embodiments and methods within the scope and spirit of thedisclosure.

All documents referenced herein are hereby incorporated by reference.

We claim:
 1. A method of processing a signal, comprising: taking asignal formed from a plurality of source signal emitters and expressedin an original domain; decomposing the signal into a mathematicalrepresentation of a plurality of constituent elements in an alternatedomain; analyzing the plurality of constituent elements to associate atleast a subset of the constituent elements with at least one of theplurality of source signal emitters; separating at least a subset of theconstituent elements based on the association; and reconstituting atleast a subset of constituent elements to produce an output signal in atleast one of the original domain, the alternate domain and anotherdomain.
 2. The method of claim 1 wherein analyzing further comprisesusing direction-of-arrival estimates within a unified domainrepresentation.
 3. The method of claim 1 wherein the analyzing theplurality of the constituent elements further comprises using complexspectral phase evolution of at least a subset of the constituentelements.
 4. The method of claim 1 wherein separating the constituentelements further comprises associating at least a subset of constituentelements with a tracklet.
 5. The method of claim 4 wherein separatingthe constituent elements further comprises grouping tracklets thatemanate from a single source.
 6. A method of processing a time domainsignal, comprising: receiving an input signal comprising a time domainsignal stream and creating a first windowed data set and a secondwindowed data set from the signal stream, wherein an initiation of thesecond windowed data set time lags an initiation of the first windoweddata set; converting the first windowed data set and the second windoweddata set to a frequency domain and storing the resulting data asfrequency domain data having a fundamental transform resolution;performing complex spectral phase evolution (CSPE) on the frequencydomain data to estimate component frequencies of the first and secondwindowed data sets at a resolution greater than the fundamentaltransform resolution; using the component frequencies estimated in theCSPE, sampling a set of frequency-domain high resolution windows toselect a frequency-domain high resolution window that fits at least oneof the amplitude, phase, amplitude modulation and frequency modulationof a component of an underlying signal wherein the component comprisesat least one oscillator peak; using a tracking algorithm to identify atleast one tracklet comprised of one or more oscillator peaks thatemanate from a single oscillator source within the underlying signal;grouping tracklets that emanate from single sources; and providing anoutput signal.
 7. The method of claim 6 wherein the input signalcomprises an analog voice signal.
 8. The method of claim 6 wherein thesignal is reconstructed to reproduce a conversation of one or moreselected speakers.
 9. The method of claim 6 wherein the input signal isreceived at a hearing aid and the output is delivered ultimately to anear.
 10. The method of claim 6 wherein the set of frequency-domain highresolution windows is pre-computed and stored.
 11. The method of claim 6wherein the set of frequency-domain high resolution windows is computedin one of near real time and real time.
 12. The method of claim 11wherein the set of frequency-domain high resolution windows iscalculated from a functional representation.
 13. The method of claim 6further comprising providing a unified domain representation of the datafrom the CSPE to enable use of direction-of-arrival estimates in theunified domain to assist in fitting to a constituent component of thesignal.
 14. The method of claim 6, further comprising adding at least aportion of constituent elements not associated with a desired signalsource to provide a modified level of background noise along with apreserved portion of the original input signal, including when thepreserved portion is enhanced or increased in amplitude.
 15. The methodof claim 14 wherein the preserved portion is increased in amplitude. 16.A method of processing a time domain signal, comprising: receiving atime domain signal stream and creating a first windowed data set and asecond windowed data set from the signal stream, wherein an initiationof the second windowed data set time lags an initiation of the firstwindowed data set; converting the first windowed data set and the secondwindowed data set to a frequency domain and storing the resulting dataas frequency domain data having a fundamental transform resolution;performing complex spectral phase evolution (CSPE) on the frequencydomain data to estimate component frequencies of first and second windowat a resolution greater than the fundamental transform resolution; usingthe component frequencies estimated in the CSPE, sampling a set ofstored high resolution frequency-domain windows in a singlettransformation process to select a high resolution frequency-domainwindow that fits at least one of the amplitude, phase, amplitudemodulation and frequency modulation of the underlying signal oscillator;storing the parameters required for at least one of FM creation and AMcreation in the frequency domain, wherein the parameters for FM creationinclude amplitude, phase, reference frequency, and modulation rate andthe parameters for AM creation include amplitude, phase, frequency, andamplitude envelope information; and recreating the frequency spectrumfor at least one of an FM-modulating oscillator peak and anAM-modulating oscillator peak, such frequency spectrum including anytransient effects where the oscillator may turn on or off at some pointwithin the data sample window.
 17. The method of claim 16 furthercomprising combining a recreated frequency spectrum for at least one ofan FM-modulating oscillator peak and an AM-modulating oscillator peakwith a frequency spectrum for another oscillator peak.
 18. The method ofclaim 16 further comprising, including in the frequency spectrumrepresentation of an oscillator peak signal spreading information torepresent the signal element to a desired degree of accuracy.
 19. Themethod of claim 16 further comprising, selecting analysis windows forthe frequency spectrum representation appropriate to account forsmearing through at least one of the zero frequency bin and the highestfrequency bin.
 20. The method of claim 16 further comprising includingin the frequency spectrum of an oscillator peak signal spreadinginformation to account for at least one of a continuous wrapping effectand a plurality of wrapping effects.